Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Loop for row-wise averaging of multiple files using awk

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Loop for row-wise averaging of multiple files using awk
# 1  
Old 02-16-2012
Loop for row-wise averaging of multiple files using awk
Hello all, I need to compute a row-wise average of files with a single column based on the pattern of the filenames. I really appreciate any help on this. it would just be very difficult to do them manually as the rows are mounting to 100,000 lines. the filenames are as below with convention as 2000-year; 01-month; 01-day.

Code:
20000101.txt 20000108.txt 20000115.txt
20000102.txt 20000109.txt 20000116.txt
20000103.txt 20000110.txt 20000117.txt
20000104.txt 20000111.txt 20000118.txt
20000105.txt 20000112.txt 20000119.txt
20000106.txt 20000113.txt 20000120.txt
20000107.txt 20000114.txt 20000121.txt

sample contents of individual files are like:

Code:
file 1: file 2: file 7:    outfile:
2        4        2        (2+4+2)/3
4        5        2        (4+5+2)/3
3        6        3        (3+6+3)/3
5        7        4        (5+7+4)/3
n1       n2       n3       (n1+n2+n3)/3

I want to compute for the average of each row in the files from day 01-07 of month 01; day 08-14 of month 01; day 15-21 of month 01; day 22-31 of month 01; day 01-07 of month 02; day 08-14 of month 02; and so on for the rest of the daily files in my directory and save the outputs on separate weekly files.

what i have in mind is something like a pseudo-code below:

Code:
for file in *.txt
do 
   scan filenames based on month-day pattern
   compute for average of day from 1-7 for month 01 > 200001wk01.dat
   compute for average of day from 8-14 for month 01 > 200001wk02.dat
   compute for average of day from 15-21 for month 01 > 200001wk03.dat
   compute for average of day from 22-31 for month 01 > 200001wk04.dat
   ...to weekly average of month 12
done

thank you very much.
# 2  
Old 02-16-2012
If you do it that way, you would need to sort the file first so that they are in the right order with a sort because you need the input from 7 files and 7-10 files in the last week of the months at the same time.
Code:
ls | sort -n -k1.1,1.4 -k1.5,1.6 -k1.7,1.8 |
while read file
do

But that could get a bit complicated. Perhaps you would be better off enumerating the weeks and months in loops and reading the corresponding files from the directory...
Another route might be to consolidate the information from the files into arrays and process them at the end..
Last edited by Scrutinizer; 02-16-2012 at 02:35 AM..
# 3  
Old 02-16-2012
thanks for your suggestions. If I try consolidating the files into arrays, how do I get with computing for the average for each row?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Search string in multiple files and display column wise

I have 3 files. Each of those files have the same number of records, however certain records have different values. I would like to grep the field in ALL 3 files and display the output with only the differences in column wise and if possible line number File1 Name = Joe Age = 33... (3 Replies)
Discussion started by: sidnow
3 Replies

2. UNIX for Dummies Questions & Answers

Print row wise

Hi Help, I have an I/p, which looks like --- FF GG HH I want the o/p to be like --- FF GG HH. How we can do that? Thanks (7 Replies)
Discussion started by: Indra2011
7 Replies

3. Shell Programming and Scripting

How to use a loop for multiple files in a folder to run awk command?

Dear folks I have two data set which there names are "final.map" and "1.geno" and look like this structures: final.map: gi|358485511|ref|NC_006088.3| 2044 gi|358485511|ref|NC_006088.3| 2048 gi|358485511|ref|NC_006088.3| 2187 gi|358485511|ref|NC_006088.3| 17654 ... (2 Replies)
Discussion started by: sajmar
2 Replies

4. Shell Programming and Scripting

FOR loop with multiple files as input and awk

Hi all , i want to pass multiple files as input to a for loop for i in file1 file2 file3 do some awk action < $i >> $i.out done but im getting error in that for loop is the way i use to pass files to awk using for correct and 2.we can directly pass multiple files to awk as... (7 Replies)
Discussion started by: zozoo
7 Replies

5. Shell Programming and Scripting

calculating row-wise standard deviation using awk

Hi, I have a file containing 100,000 rows-by-120 columns and I need to compute for the standard deviation for each row. Any idea on how to calculate row-wise standard deviation using awk? My sample data looks like this: input data: 23 35 12 25 16 17 18 19 29 12 12 26 15 14 15 23 12 12... (2 Replies)
Discussion started by: ida1215
2 Replies

6. Shell Programming and Scripting

Averaging each row with null values

Hi all, I want to compute for the average of a file with null values (NaN) for each row. any help on how to do it. the sample file looks like this. 1.4 1.2 1.5 NaN 1.6 1.3 1.1 NaN 1.3 NaN 2.4 1.3 1.5 NaN 1.5 NaN 1.2 NaN 1.4 NaN I need to do a row-wise averaging such that it will sum only... (14 Replies)
Discussion started by: ida1215
14 Replies

7. Shell Programming and Scripting

averaging specific column of multiple files

Hi all, I'm needing help again on scripting. I have weekly files with 3 columns, and I need to do monthly averaging on the values on column 3, the file naming convention is as follows: 20000105.u- 2000:year 01:month 05:day 20000112.u 20000119.u 20000126.u 20000202.u 20020209.u I need to... (15 Replies)
Discussion started by: ida1215
15 Replies

8. Shell Programming and Scripting

Averaging 3 files with multiple rows

Hi, I am trying to average the values from 3 files with the same format. They are very large files so I will describe the file and show some it of. Basically the file has 83 columns (with nearly 7000 rows). The first three columns are the same for each file while the remaining 80 are values that... (1 Reply)
Discussion started by: kylle345
1 Replies

9. Shell Programming and Scripting

Combine Multiple text or csv files column-wise

Hi All I am trying to combine columns from multiple text files into a single file using paste command but the record length being unequal in the different files the data is running over to the closest empty cell on the left. Please see below. What can i do to resolve this ? File 1 File... (15 Replies)
Discussion started by: venky_ibm
15 Replies

10. Shell Programming and Scripting

awk-gsub on column-wise on each row

awk '{ gsub(/....=/,""); print }' want.dat >final.dat the above awk command which removes all the chars before and including '=' on the entire row. --thats what it meant be.:) but i need to remove text on column-wise on each row. many thanks, EM ---------- Post updated at 10:00 AM... (4 Replies)
Discussion started by: elamurugu
4 Replies
Login or Register to Ask a Question