Shell Programming and Scripting

Hey everyone,

So I have a task that I want to complete with awk (+ find, or something similar), but can't quite achieve it by myself...

I have 60 GB of files that I want to modify. They each consist of 2 columns of numbers, with up to 50,000 lines in a file.
e.g.

Code:

   1.607743       5.928237    
   1.578281       5.941602    
   1.480728       5.867404    
   1.592771       5.980338   
    ....               ....

The files are in subdirectories one layer down from the parent, and look like e.g.

Code:

322_b3.447/resu_1_1_0
322_b3.447/resu_2_1_0

I want to reduce the file sizes by averaging the numbers in the columns in groups of 10, and creating new files with names e.g. smaller/322_b3.447/resu_1_1_0

I've tried to make a do loop with general structure:

Code:

for file in $(find -type f -name 'resu_*')
do 
   some command to specify "$newfile"
   awk '
   {s1+=$1;s2+=$2}
   !(NR%10){
          {
       print (s1/10,s2/10) >  "$newfile"
     }
     s1=0; s2=0
   }' $file
done

However, I have trouble specifying the name of the new file because 'find' returns for $file names like ./resu_2_1_0, and I don't know how to remove the "./" and include a different directory there.

Also, my awk loop only works when the number of lines per file is exactly a multiple of 10. Otherwise the output for the remaining, say, 7 lines, will still be divided by 10 and hence be too small.

I would be extremely grateful if anyone can help me with this - I'm out of my depth!

You could try something like this:

Code:

find -type f -name 'resu_*' |
  while IFS= read -r; do
    _newd=smaller/${REPLY%/*}
    _newn=smaller/$REPLY
    [ -d "$_newd" ] || mkdir -p "$_newd"  
    awk > "$_newn" 'END { 
      if (f) print f/10, s/10 
      }
    !(NR%10) { 
      print f/10, s/10  
      f = s = 0
      }
    { f += $1; s += $2 }' "$REPLY"
  done

You should add exception handling.
You could also try to improve the script using xargs in order to reduce the number of calls made to awk (in this case the awk code should be modified also).
On Solaris you should use gawk, nawk or /usr/xpg4/bin/awk.

Why are you using find?

Code:

for file  in */resu_*

Code:

do 
   newfile=${file#*/}    ## this removes the directory; adjust to taste
   awk '
   {s1+=$1;s2+=$2}
   !(NR%10){
          {
       print (s1/10,s2/10)
     }
     s1=0; s2=0
   }' "$file"  >  "$newfile"
done

It will not print anything for a trailing fraction of 10 lines. You need to add an END block to calculate the last amount, if any.

Good point!

Thanks radoulov and cfajohnson. I've been playing around with your code and I think that I basically have what I need. The first bit of code from radoulov gave me the file structure that I need, but seems to take the 1st 9 records and divide them by 10, then goes in blocks of 10 thereafter, leaving 1 remaining record (for my test file with 100 lines).

I think that the following combination works. Are there any landmines in this that I should watch out for?

Code:

for file  in */resu_*
do 
    _newd=smaller/${file%/*}
    _newn=smaller/$file
    [ -d "$_newd" ] || mkdir -p "$_newd" 

   awk '
   {s1+=$1;s2+=$2}
   !(NR%10){
          {
       print (s1/10,s2/10)
     }
     s1=0; s2=0
   }' "$file"  >  "$_newn"
done

This could happen if you have some kind of header (or simply a newline) before the first record.