Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to calculate mean in AWK? line by line several files, thousands of lines

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to calculate mean in AWK? line by line several files, thousands of lines
# 1  
Old 05-26-2011
Error How to calculate mean in AWK? line by line several files, thousands of lines
I'm kinda stuck on this one, I have 7 files with 30.000 lines/file like this
Code:
050	0.023	0.504336
050	0.024	0.529521
050	0.025	0.538908
050	0.026	0.537035

I want to find the mean line by line of the third column from the files named like this:
Code:
Stat-f-1.dat .... Stat-f-7.dat 
Stat-s-1.dat .... Stat-s-7.dat

# 2  
Old 05-26-2011
Calculate the mean of third column in awk:
Code:
awk '{x+=$3;next}END{print x/NR}' data

Now, do you want the mean from values of each files separately?
Or all Stat-f* files together, then all Stat-s*...?
# 3  
Old 05-26-2011
I wanted the average line by line, 1st line of all files print average in a new file, 2nd line, and so on...
# 4  
Old 05-26-2011
not tested....
Code:
awk '{x[FNR]+=$3;next}END{for(i=1;i in x;i++) print x[i]/ARGC}' allMyFilesWildWildCard

This User Gave Thanks to vgersh99 For This Post:
AriasFco
# 5  
Old 05-26-2011
file 1
Code:
001     0.046   0.667267
001     0.047   0.672028
001     0.048   0.656025
001     0.049   0.660557

002     0.000   0.669553
002     0.001   0.594648
002     0.002   0.586738
002     0.003   0.593728
002     0.004   0.593658

File 2
Code:
001     0.046   0.654565
001     0.047   0.665057
001     0.048   0.660074
001     0.049   0.670424

002     0.000   0.669462
002     0.001   0.594793
002     0.002   0.589329
002     0.003   0.593949
002     0.004   0.592371

They are seven (7) files

Desired Output
Code:
001     0.046   Average of the nth line from the two files
001     0.047   Average of the nth line from the two files
001     0.048   Average of the nth line from the two files
001     0.049   Average of the nth line from the two files

002     0.000   Average of the nth line from the two files
002     0.001   Average of the nth line from the two files
002     0.002   Average of the nth line from the two files
002     0.003   Average of the nth line from the two files
002     0.004   Average of the nth line from the two files


Thanks for your help!! Can you try on those files?

The filenames are like Stat-zz.dat, zz goes from 00 to 07
# 6  
Old 05-26-2011
Quote:
Originally Posted by AriasFco
file 1
Code:
001     0.046   0.667267
001     0.047   0.672028
001     0.048   0.656025
001     0.049   0.660557

002     0.000   0.669553
002     0.001   0.594648
002     0.002   0.586738
002     0.003   0.593728
002     0.004   0.593658

File 2
Code:
001     0.046   0.654565
001     0.047   0.665057
001     0.048   0.660074
001     0.049   0.670424

002     0.000   0.669462
002     0.001   0.594793
002     0.002   0.589329
002     0.003   0.593949
002     0.004   0.592371

They are seven (7) files

Desired Output
Code:
001     0.046   Average of the nth line from the two files
001     0.047   Average of the nth line from the two files
001     0.048   Average of the nth line from the two files
001     0.049   Average of the nth line from the two files

002     0.000   Average of the nth line from the two files
002     0.001   Average of the nth line from the two files
002     0.002   Average of the nth line from the two files
002     0.003   Average of the nth line from the two files
002     0.004   Average of the nth line from the two files

Thanks for your help!! Can you try on those files?

The filenames are like Stat-zz.dat, zz goes from 00 to 07
I'll let you do the honors and let us know how it goes!
Good luck!
# 7  
Old 05-26-2011
@vgersh99: you need to divide by (ARGC-1)

@AriasFco: try this:
Code:
$ awk '{
  avg[$1,$2]+=$3; 
  i[FNR]=$1 SUBSEP $2
}
END{
  for(k in i) sorted_i[j++]=k+0; 
  n=asort(i,sorted_i); 
  for(j=1; j<=n; j++) 
    print i[j]"\t" avg[i[j]]/(ARGC-2)
}' SUBSEP='\t' Stat-f-*.dat

These 2 Users Gave Thanks to mirni For This Post:
AriasFco vgersh99
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Get an output of lines in pattern 1st line then 10th line then 11th line then 20th line and so on.

Input file: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 (6 Replies)
Discussion started by: Sagar Singh
6 Replies

2. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies

3. Shell Programming and Scripting

awk to combine lines from line with pattern match to a line that ends in a pattern

I am trying to combine lines with these conditions: 1. First line starts with text of "libname VALUE db2 datasrc" where VALUE can be any text. 2. If condition1 is met then continue to combine lines through a line that ends with a semicolon. 3. Ignore case when matching patterns and remove any... (5 Replies)
Discussion started by: Wes Kem
5 Replies

4. Shell Programming and Scripting

Need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line...

Hello, I need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line... An example of entries in the file would be: SRVXPAPI001 ERRO JUN24 07:28:34 1775 REASON= 0000, PROCID= #E506 #1065: TPCIPPR, INDEX= 003F ... (8 Replies)
Discussion started by: Ferocci
8 Replies

5. Shell Programming and Scripting

How to read a two files, line by line in UNIX script and how to assign shell variable to awk ..?

Input are file and file1 file contains store.bal product.bal category.bal admin.bal file1 contains flip.store.bal ::FFFF:BADC:CD28,::FFFF:558E:11C5,6,8,2,1,::FFFF:81C8:CA8B,::FFFF:BADC:CD28,1,0,0,0,::FFFF:81C8:11C5,2,1,0,0,::FFFF:81DC:3111,1,0,1,0 store.bal.... (2 Replies)
Discussion started by: veeruasu
2 Replies

6. Shell Programming and Scripting

read line by line and calculate the co-presence of variables

Hey guyz, I have a table which shows the presence or absence of my variables (A,B,C,...) in my observations (1,2,3,...) * A B C ... 1 1 0 1 2 1 1 0 3 1 0 0 ... I want to calculate the co-presence of my variables. to have a table shows the pairwise presence of the variables (have... (1 Reply)
Discussion started by: @man
1 Replies

7. Shell Programming and Scripting

how to read the contents of two files line by line and compare the line by line?

Hi All, I'm trying to figure out which are the trusted-ips and which are not using a script file.. I have a file named 'ip-list.txt' which contains some ip addresses and another file named 'trusted-ip-list.txt' which also contains some ip addresses. I want to read a line from... (4 Replies)
Discussion started by: mjavalkar
4 Replies

8. Shell Programming and Scripting

add number in lines line by line in different files

I have a set of log files that are in the following format ======= set_1 ======== counter : 315 counter2: 204597 counter3: 290582 ======= set_2 ======== counter : 315 counter2: 204597 counter3: 290582 ======= set_3 ======== counter : 315 counter2: 204597 counter3: 290582 Is... (6 Replies)
Discussion started by: grandguest
6 Replies

9. Shell Programming and Scripting

Input 2 files, calculate diffs line by line

Hi I am new to Unix and need help with the following (to you all I'm sure) simple task. I am trying to output the differences between previous snaphots of various filesystem sizes to the present sizes. I have three files (e.g.) : file 1 file 2 file 3 10 100 /var... (4 Replies)
Discussion started by: bigbuk
4 Replies
Login or Register to Ask a Question