You said you had a large number of files. My hope is that we'll be able to speed it up enough that you will be able to process each file in a reasonable amount of time and you can do your parallelization by processing different files on different CPUs instead of different parts of single files on different CPUs.

A simple:

Code:

        if [ -z "$x" ]
        then    continue
        fi
            or
        [ -z "$x" ] && continue

shouldn't take long in your script (and won't be needed in an awk script) and will avoid the diagnostic messages from the failed comparisons in bc. The cost of starting up bc once when you know it won't work will easily take longer than running the above code in the shell several dozen (or more likely, several hundred or several thousand) times.

The awk utility (and 1993 or later versions of the Korn shell) use double-precision floating point arithmetic which gives you 15 to 17 decimal digits of precision. If you sum up the squares of 1,000,001 numbers in the range 3.32E+001 (1102.24) to 6.37E+001 (4057.69) (those were the biggest and smallest numbers in field 5 of your sample input in the 1st message in this thread) that could be up to 40576904057.69 which is 15 digits. If you have a number with a larger exponent (e.g., 9.87E+010) and a number with a smaller exponent (e.g., 1.23E-008) you will easily exceed double-precision limitations with just those two numbers. Having seen only three lines of your input data, I can't begin to guess whether awk can do what you need or whether you'll need to use awk to feed a stream of values to be added together by bc with as many digits preserved after the decimal point as you desire. Either way, you can do it with a single invocation of awk and no more than one invocation of bc to process an entire input file. And, awk should be able to do the conversion of exponential notation to a digit string rather than needing to ask bc to multiply by powers of 10.