There aren't 3 numbers in. There are, in fact, millions and millions. The 3 lines that I posted are just a sample from a 9,000,000 line file. What I'm trying to do is use all of a system's CPU to complete a task faster. So, let me explain a little differently.

I have a huge file. Normally, awk would just read from line 1 to the last line, using a single CPU core. This script gives me what I want:

Code:

#!/usr/local/bin/mawk -f

    {
    for (i = 1; i <= NF; i++)
        {
        if ($i ~ /^[0-9]+$/)
            {
            SUM += $i
            }
        }
    }
END {
    printf "%.0f\n",  SUM
    }

If I run that script on my data file, it works just fine, albeit slowly, as it uses only a single core. So, I decided to instead make 7 copies of the script and use them all at the same time. Those are called proc.1.mawk, proc.2.mawk ... proc.7.mawk. Another mawk script is feeding lines into those scripts. Lines where NR%7 is 0 go to the first proc script, those where NR%7 is 1 go to the second proc script, and so on so that those where NR%7 is 6 go to the last proc script. That mawk script is simply this:

Code:

awk -F , -v count=7 'NR > 1 {mod = NR%count + 1 ; proc = "./proc." mod ".mawk -F ,"; print $0 | proc}' datafile.csv

I call that the "distributor". You can see that it (1) calculates NR%7 for each line, (2) figures out which processor script should be handling that line, and (3) pipes the line to that processor script. So, each proc script is doing 1/7th of the total lines. That all works properly and, in fact, shows a huge improvement in processing time.

The problem is simply that each of those proc scripts write their output to stdout. So, I have to do something like this:

Code:

awk -F , -v count=7 'NR > 1 {mod = NR%count + 1 ; proc = "./proc." mod ".mawk -F ,"; print $0 | proc}' datafile.csv | awk '{SUM += $1} END {print SUM}'

That works just fine, but I'd rather have summing occur in the same distributor that passes the lines to the processor scripts. I tried something like this:

Code:

awk -F , -v count=7 'NR > 1 {mod = NR%count + 1 ; proc = "./proc." mod ".mawk -F ,"; SUM += (print $0 | proc)}' datafile.csv

That fails with a syntax error.

So, is there a way to get the distributor to receive back the output of the proc.NUM.mawk scripts and sum them together?