Shell Programming and Scripting

Thank you drl ...

Pardon my ignorance, from your post I can follow that the code can break up up the input file to feed into a number of parallel CPU operations,, and due to parallel processing, the real time reduces significantly.. You are right that most of your post is too advanced for me to understand,

can I utilize your code directly along with Chublers to do the parallel processing without the help of a code-literate colleague ? You have mentioned you saved Chublers code in the file a1, then save your script as s1.sh and run ./s1 ?

Hi.
My post was to illustrate the idea of parallelizing the solution. In order for this to become a production code, the perl code parallel (or something similar) needs to be available. Is it on your system?

Then for each section of data, your header -- "Org1 Org2 Org3 .... Org750" needs to be present at the beginning. Do you know how that might be done?

The hardware system needs to have more than one core (CPU) available. What hardware, OS, are you using?

The final results from command parallel would need to be collected and summed for the same counts from different instances. Can you do that, based on how the code of Chubler produces output?

If I were designing such a production code, I would want to be able to run tests on smaller sections of the data so that I could be sure that all parts of the tasks are correct, and that the entire process is near optimal and balanced (IO and CPU) for the platform, hardware, OS, utilitiies.

Best wishes ... cheers, drl

This code should split the file up into Pieces and run the solution in parallel.

Set CPUS= to the number of parallel instances you would like to run (no more than the number of CPUs you have in the system), and SOURCE= to your source datafile name.

While this is running the passN.log files will have their % complete.

Code:

CPUS=4
SOURCE=og
lines=$(wc -l < $SOURCE)
end=0
i=1

while [ $i -le $CPUS ]
do
   ((start=end+1))
   ((end=lines*i/CPUS))
   awk -v s=$start -v e=$end 'NR==1||((NR>=s&&NR<=e))' $SOURCE > pass$i
   ((i++))
done

i=0
while [ $i -le $CPUS ]
do
    nohup awk -vL=$((lines/CPUS)) 'NR==1{for(cols=1;cols<=NF;cols++)H[cols]=$cols;next}
    {for(i=1;i<=NF;i++)
        for(j=i+1;j<=NF;j++)
           if($i==$j) c[i,j]++
    }
    P!=int(NR/L*100) {
       P=int(NR/L*100);
       printf("Processed %d%%%c", P, 13) > "/dev/stderr"
    }
    END{
       for(i=1;i<cols;i++)
          for(j=i+1;j<cols;j++)
          print H[i],H[j],0+c[i,j]
    } ' pass$i > pass$i.out 2> pass$i.log &
    ((i++))
done

Once the have finished you will have passN.out data files that can be summed to give your final totals with:

Code:

awk '
   {L[FNR]=$1 " " $2;N[FNR]+=$3}
   END{for(i=1;i<=FNR;i++) print L[i],N[i]}' pass[0-9]*.out > result.txt

As drl pointed out you should really prove this first with a smaller data file (eg 1000 lines or so) before investing many days of CPU processing on the full sized dataset.