Shell Programming and Scripting

Put this code into "script.awk":

Code:

NR==1{
  gsub(" +","\t")
  print
}
NR>1&&(NR-1)%360{
  for (i=3;i<=NF;i++){
    a[i]+=$i
  }
}
NR>1&&!((NR-2)%360){
  t=$1"\t"$2"\t"
}
NR>1&&!((NR-1)%360){
  printf t
  for (i=3;i<=NF;i++){
    printf "%.10e\t",a[i]/360
    a[i]=0
  }
  printf "\n"
}

Then run it like that:

Code:

awk -f script.awk input > output

It will change the format of some columns, and it will change the field delimiter to single TAB as keeping all the spaces intact is quite troublesome.

Hey Bartus,

for the averaging code you gave me I'm unsure as to what the code actually does, can you just breifly run through what the parts of the script do?

Code:

#!/bin/gawk -f

# this script is used to average the data into 60 minute intervals


NR==1{
  gsub(" +","\t")
  print
}
NR>1&&(NR-1)%720{
  for (i=3;i<=NF;i++){
    a[i]+=$i
  }
}
NR>1&&!((NR-2)%720){
  t=$1"\t"$2"\t"
}
NR>1&&!((NR-1)%720){
  printf t
  for (i=3;i<=NF;i++){
    printf "%.5e\t",a[i]/720
    a[i]=0
  }
  printf "\n"
}

When analyzing my code I noticed that it is not giving accurate output... This should calculate the average properly:

Code:

NR==1{
  gsub(" +","\t")
  print
}
NR>1&&(NR-1)%720{
  for (i=3;i<=NF;i++){
    a[i]+=$i
  }
}
NR>1&&!((NR-2)%720){
  t=$1"\t"$2"\t"
}
NR>1&&!((NR-1)%720){
  for (i=3;i<=NF;i++){
    a[i]+=$i
  }
  printf t
  for (i=3;i<=NF;i++){
    printf "%.5e\t",a[i]/720
    a[i]=0
  }
  printf "\n"
}

In simple words, it is going through whole file and checks line numbers:
for line 1 it is printing the header;
for line 2 it is saving time into variable
for lines 2-720 it is summing the columns in appropriate array elements;
for line number 721 it is adding the last value of the range to the array elements, then it prints the time saved at line 2 and prints calculated average value.
Using modulo operator causes this steps to be performed again for line 722-1440, 1441 etc.

Hi gd9629
I recognise the data you're working with (measurements of atmospheric carbon dioxide, methane and water vapour) - my day job is running instruments such as you have.

A couple of comments if I may;
It would be fairly standard procedure to average such data to a fixed timestamp rather than have it defined by the start of the data record eg hourly averages from minute 0 to minute 59. This simplifies comparison or combination of different data sets and takes care of pesky issues such as periods of missing data

You can reduce your computation time by ignoring the data which are either pointless or meaningless to average. PM me if you want to discuss further as that discussion would be very much off-topic

My own approach would be to reduce the data to a one minute averages, filtered vs the diagnostics information in $6 to $15 (again PM me if you want further clarification) and written to a daily or monthly file. You can then calculate averages and other stats for whatever periods you wish without having to revisit the rather large raw data files you started with (must be at least 350Mb per day?). You can also reformat to whatever suits you. I'd go with space or comma delimited and lose the exponents and a lot of excess digits

cheers