Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Loop for row-wise averaging of multiple files using awk
Top Forums Shell Programming and Scripting Loop for row-wise averaging of multiple files using awk Post 302598985 by ida1215 on Thursday 16th of February 2012 01:08:51 AM
Old 02-16-2012
Loop for row-wise averaging of multiple files using awk
Hello all, I need to compute a row-wise average of files with a single column based on the pattern of the filenames. I really appreciate any help on this. it would just be very difficult to do them manually as the rows are mounting to 100,000 lines. the filenames are as below with convention as 2000-year; 01-month; 01-day.

Code:
20000101.txt 20000108.txt 20000115.txt
20000102.txt 20000109.txt 20000116.txt
20000103.txt 20000110.txt 20000117.txt
20000104.txt 20000111.txt 20000118.txt
20000105.txt 20000112.txt 20000119.txt
20000106.txt 20000113.txt 20000120.txt
20000107.txt 20000114.txt 20000121.txt

sample contents of individual files are like:

Code:
file 1: file 2: file 7:    outfile:
2        4        2        (2+4+2)/3
4        5        2        (4+5+2)/3
3        6        3        (3+6+3)/3
5        7        4        (5+7+4)/3
n1       n2       n3       (n1+n2+n3)/3

I want to compute for the average of each row in the files from day 01-07 of month 01; day 08-14 of month 01; day 15-21 of month 01; day 22-31 of month 01; day 01-07 of month 02; day 08-14 of month 02; and so on for the rest of the daily files in my directory and save the outputs on separate weekly files.

what i have in mind is something like a pseudo-code below:

Code:
for file in *.txt
do 
   scan filenames based on month-day pattern
   compute for average of day from 1-7 for month 01 > 200001wk01.dat
   compute for average of day from 8-14 for month 01 > 200001wk02.dat
   compute for average of day from 15-21 for month 01 > 200001wk03.dat
   compute for average of day from 22-31 for month 01 > 200001wk04.dat
   ...to weekly average of month 12
done

thank you very much.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk-gsub on column-wise on each row

awk '{ gsub(/....=/,""); print }' want.dat >final.dat the above awk command which removes all the chars before and including '=' on the entire row. --thats what it meant be.:) but i need to remove text on column-wise on each row. many thanks, EM ---------- Post updated at 10:00 AM... (4 Replies)
Discussion started by: elamurugu
4 Replies

2. Shell Programming and Scripting

Combine Multiple text or csv files column-wise

Hi All I am trying to combine columns from multiple text files into a single file using paste command but the record length being unequal in the different files the data is running over to the closest empty cell on the left. Please see below. What can i do to resolve this ? File 1 File... (15 Replies)
Discussion started by: venky_ibm
15 Replies

3. Shell Programming and Scripting

Averaging 3 files with multiple rows

Hi, I am trying to average the values from 3 files with the same format. They are very large files so I will describe the file and show some it of. Basically the file has 83 columns (with nearly 7000 rows). The first three columns are the same for each file while the remaining 80 are values that... (1 Reply)
Discussion started by: kylle345
1 Replies

4. Shell Programming and Scripting

averaging specific column of multiple files

Hi all, I'm needing help again on scripting. I have weekly files with 3 columns, and I need to do monthly averaging on the values on column 3, the file naming convention is as follows: 20000105.u- 2000:year 01:month 05:day 20000112.u 20000119.u 20000126.u 20000202.u 20020209.u I need to... (15 Replies)
Discussion started by: ida1215
15 Replies

5. Shell Programming and Scripting

Averaging each row with null values

Hi all, I want to compute for the average of a file with null values (NaN) for each row. any help on how to do it. the sample file looks like this. 1.4 1.2 1.5 NaN 1.6 1.3 1.1 NaN 1.3 NaN 2.4 1.3 1.5 NaN 1.5 NaN 1.2 NaN 1.4 NaN I need to do a row-wise averaging such that it will sum only... (14 Replies)
Discussion started by: ida1215
14 Replies

6. Shell Programming and Scripting

calculating row-wise standard deviation using awk

Hi, I have a file containing 100,000 rows-by-120 columns and I need to compute for the standard deviation for each row. Any idea on how to calculate row-wise standard deviation using awk? My sample data looks like this: input data: 23 35 12 25 16 17 18 19 29 12 12 26 15 14 15 23 12 12... (2 Replies)
Discussion started by: ida1215
2 Replies

7. Shell Programming and Scripting

FOR loop with multiple files as input and awk

Hi all , i want to pass multiple files as input to a for loop for i in file1 file2 file3 do some awk action < $i >> $i.out done but im getting error in that for loop is the way i use to pass files to awk using for correct and 2.we can directly pass multiple files to awk as... (7 Replies)
Discussion started by: zozoo
7 Replies

8. Shell Programming and Scripting

How to use a loop for multiple files in a folder to run awk command?

Dear folks I have two data set which there names are "final.map" and "1.geno" and look like this structures: final.map: gi|358485511|ref|NC_006088.3| 2044 gi|358485511|ref|NC_006088.3| 2048 gi|358485511|ref|NC_006088.3| 2187 gi|358485511|ref|NC_006088.3| 17654 ... (2 Replies)
Discussion started by: sajmar
2 Replies

9. UNIX for Dummies Questions & Answers

Print row wise

Hi Help, I have an I/p, which looks like --- FF GG HH I want the o/p to be like --- FF GG HH. How we can do that? Thanks (7 Replies)
Discussion started by: Indra2011
7 Replies

10. Shell Programming and Scripting

Search string in multiple files and display column wise

I have 3 files. Each of those files have the same number of records, however certain records have different values. I would like to grep the field in ALL 3 files and display the output with only the differences in column wise and if possible line number File1 Name = Joe Age = 33... (3 Replies)
Discussion started by: sidnow
3 Replies

LEARN ABOUT REDHAT

psc

PSC(1)							      General Commands Manual							    PSC(1)

NAME

       psc - prepare sc files

SYNOPSIS

       psc [-fLkrSPv] [-s cell] [-R n] [-C n] [-n n] [-d c]

DESCRIPTION

       Psc  is used to prepare data for input to the spreadsheet calculator sc(1).  It accepts normal ascii data on standard input.  Standard out-
       put is a sc file.  With no options, psc starts the spreadsheet in cell A0.  Strings are right justified.  All data on a line is entered	on
       the  same row; new input lines cause the output row number to increment by one.	The default delimiters are tab and space.  The column for-
       mats are set to one larger than the number of columns required to hold the largest value in the column.

OPTIONS

       -f     Omit column width calculations.  This option is for preparing data to be merged with an existing spreadsheet.  If the option is  not
	      specified, the column widths calculated for the data read by psc will override those already set in the existing spreadsheet.

       -L     Left justify strings.

       -k     Keep  all  delimiters.   This  option  causes the output cell to change on each new delimiter encountered in the input stream.   The
	      default action is to condense multiple delimiters to one, so that the cell only changes once per input data item.

       -r     Output the data by row first then column.  For input consisting of a single column, this option will result in  output  of  one  row
	      with multiple columns instead of a single column spreadsheet.

       -s cell
	      Start  the  top  left  corner  of the spreadsheet in cell.  For example, -s B33 will arrange the output data so that the spreadsheet
	      starts in column B, row 33.

       -R n   Increment by n on each new output row.

       -C n   Increment by n on each new output column.

       -n n   Output n rows before advancing to the next column.  This option is used when the input is  arranged  in  a  single  column  and  the
	      spreadsheet is to have multiple columns, each of which is to be length n.

       -d c   Use the single character c as the delimiter between input fields.

       -P     Plain numbers only.  A field is a number only when there is no imbedded [-+eE].

       -S     All numbers are strings.

       -v     Print the version of psc

SEE ALSO

       sc(1)

AUTHOR

       Robert Bond

PSC 7.16							 19 September 2002							    PSC(1)
All times are GMT -4. The time now is 02:48 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy