Shell Programming and Scripting

So, I've been playing with speeding up some analysis we do by using multiple threads of awk (actually, mawk, but code-compatible as far as I use it) on multiple CPU cores. So, I have a big data file and I have several copies of exactly the same processor script, written in mawk. I also have a "distributor" in mawk whose only job is to determine the modulus of a given line's number and use that to send the line to the appropriate processor script. Then, when that's all done, the results of the processor scripts are joined together into a final result. Here's the "distributor" line:

Code:

mawk -F , -v count=6 'NR > 1 {mod = NR%count + 1 ; proc = "./proc." mod ".mawk -F ,"; print $0 | proc}' datafile.csv

So, lines 6, 12, 18, etc will go to proc.1.mawk, 7, 13, 19 will go to proc.2.mawk, and so on. Those work very well. The problem is in gathering up those results. Currently, each proc.X.mawk script just spits out its answer to stdout, and I have to pipe to another awk to sum those. My question, then: Is there a way to have the "distributor" receive the results from the proc.X.mawk scripts and sum them itself, such as by saving to a variable that can be acted on later?

Thanks in advance.

Show the input you have and the output you want.

Thanks for the reply. The input I have is just a very simple CSV that I grabbed for testing. A sample:

Code:

00102,20150501,0001,0,CLR, ,10.00, , , ,24, ,-4.4, ,23, ,-4.9, ,21, ,-6.1, , 88, , 0, ,000, , , ,29.70, , , , , ,30.01, ,AA, , , ,29.88, 
00102,20150501,0101,0,CLR, ,10.00, , , ,22, ,-5.6, ,21, ,-6.0, ,19, ,-7.2, , 88, , 0, ,000, , , ,29.71, , , , , ,30.02, ,AA, , , ,29.89, 
00102,20150501,0201,0,CLR, ,10.00, , , ,19, ,-7.2, ,18, ,-7.5, ,17, ,-8.3, , 92, , 0, ,000, , , ,29.72, , , , , ,30.03, ,AA, , , ,29.90,

The processor scripts that are called by the distributor just go through every field of every line, and if the field contains a positive integer it sums it.

Code:

    {
    for (i = 1; i <= NF; i++)
        {
        if ($i ~ /^[0-9]+$/)
            {
            SUM += $i
            }
        }
    }
END {
    printf "%.0f\n",  SUM
    }

It's a weird, arbitrary task, I know. Its only purpose is to test the distributor/processor idea. The distributor that calls the script above is already posted. Its output looks like:

25280650967054
25280650651146
25280650873710
25280650942866
25280650844079
25280650677879
25280671120373

Each is simply the output from each of, in this case, the 7 processor scripts that were called. So, I would get that as the output of:

Code:

mawk -F , -v count=7 'NR > 1 {mod = NR%count + 1 ; proc = "./proc." mod ".mawk -F ,"; print $0 | proc}' datafile.csv

Currently, I pipe that to:

Code:

awk '{SUM += $1} END {print SUM}'

I'd like to eliminate that pipe and have the distributor receive back each processor's results and sum them itself. It might not be readily doable, but I figured someone here might have an idea. Thank you.

How are you getting 7 numbers out for 3 numbers in?

There aren't 3 numbers in. There are, in fact, millions and millions. The 3 lines that I posted are just a sample from a 9,000,000 line file. What I'm trying to do is use all of a system's CPU to complete a task faster. So, let me explain a little differently.

I have a huge file. Normally, awk would just read from line 1 to the last line, using a single CPU core. This script gives me what I want:

Code:

#!/usr/local/bin/mawk -f

    {
    for (i = 1; i <= NF; i++)
        {
        if ($i ~ /^[0-9]+$/)
            {
            SUM += $i
            }
        }
    }
END {
    printf "%.0f\n",  SUM
    }

If I run that script on my data file, it works just fine, albeit slowly, as it uses only a single core. So, I decided to instead make 7 copies of the script and use them all at the same time. Those are called proc.1.mawk, proc.2.mawk ... proc.7.mawk. Another mawk script is feeding lines into those scripts. Lines where NR%7 is 0 go to the first proc script, those where NR%7 is 1 go to the second proc script, and so on so that those where NR%7 is 6 go to the last proc script. That mawk script is simply this:

Code:

awk -F , -v count=7 'NR > 1 {mod = NR%count + 1 ; proc = "./proc." mod ".mawk -F ,"; print $0 | proc}' datafile.csv

I call that the "distributor". You can see that it (1) calculates NR%7 for each line, (2) figures out which processor script should be handling that line, and (3) pipes the line to that processor script. So, each proc script is doing 1/7th of the total lines. That all works properly and, in fact, shows a huge improvement in processing time.

The problem is simply that each of those proc scripts write their output to stdout. So, I have to do something like this:

Code:

awk -F , -v count=7 'NR > 1 {mod = NR%count + 1 ; proc = "./proc." mod ".mawk -F ,"; print $0 | proc}' datafile.csv | awk '{SUM += $1} END {print SUM}'

That works just fine, but I'd rather have summing occur in the same distributor that passes the lines to the processor scripts. I tried something like this:

Code:

awk -F , -v count=7 'NR > 1 {mod = NR%count + 1 ; proc = "./proc." mod ".mawk -F ,"; SUM += (print $0 | proc)}' datafile.csv

That fails with a syntax error.

So, is there a way to get the distributor to receive back the output of the proc.NUM.mawk scripts and sum them together?

I'm afraid you can't do that. proc, receiving its stdin from the pipe, is run by sh in a separate process with an own individual stdout. No chance to get that back into the awk script. I don't think even a FIFO would do.

Ah, well, I was afraid of that. Oh, well... so it goes. The current solution is adequate, and since I've managed to cut the processing time down from about 2 minutes to 11 seconds, I'll live with it. :-) Thanks for the reply.