Help with computing median line by line of several files.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with computing median line by line of several files.
# 1  
Old 03-25-2014
Help with computing median line by line of several files.

I have several files which look like this. The files are much longer than this, I have just cut out the data and there are hundreds of these files
Code:
file 1
 0.00   5.905 
 0.05   5.740 
 0.10   5.570 
 0.15   5.440 
 0.20   5.310 
 0.25   5.185 
 0.30   5.075 
 0.35   5.897 
 0.40   6.883 

file 2
 0.00   8.510 
 0.05   8.130 
 0.10   7.730 
 0.15   7.390 
 0.20   7.120 
 0.25   6.850 
 0.30   9.877 
 0.35   9.463 
 0.40   9.063 

file 3 
 0.00   5.515 
 0.05   5.375 
 0.10   5.300 
 0.15   5.200 
 0.20   4.710 
 0.25   5.280 
 0.30   5.147 
 0.35   5.055 
 0.40   4.968

I need to compute the median values of column 2 for each line... that is.. I want to get value of column 2 of each line from the different files and find the median of these values then go to the line 2 of each file and get values of column 2 and find the median and so on .... All the files have exactly equal length and the values of column 1 are the same for each file.

The output should contain column 1 values and the median value computed from the column 2 values of the different files
that is
Code:
 0.00   median of (5.905, 8.510, 5.515) 
 0.05   median of (5.740, 8.130, 5.375) 
 0.10   median of (5.570, 7.730, 5.300) 
 0.15   median of (5.440, 7.390, 5.200) 
 0.20   median of (5.310, 7.120, 4.710) 
 0.25   median of (5.185, 6.850, 5.280) 
 0.30   median of (5.075, 9.877, 5.147) 
 0.35   median of (5.897, 9.463, 5.055) 
 0.40   median of (6.883, 9.063, 4.968)

Thank you
malandisa
# 2  
Old 03-25-2014
Depending on the amount of memory your system has and how big/how many these files are you may have trouble getting all the data into some sort of two-dimensional array. Construct a single file with all the columns put together (similar to your sample result) in this case. You can use the "paste" utility to do so.

Notice, that to paste files they have to be sorted before. Looking at column 1 of your samples they appear to be already.

I hope this helps.

bakunin
This User Gave Thanks to bakunin For This Post:
# 3  
Old 03-25-2014
Code:
awk '{a[$1] = (a[$1] == "") ? $2 : (a[$1] "|" $2)}
  END {for(x in a)
    {split(a[x], b, "|"); n=asort(b, c);
 if(n % 2 == 1)
   {m = c[int(n /2) + 1]}
 else
   {m = (c[int(n /2)] + c[int(n /2) + 1]) / 2};
 print x, m}}' file* | sort -n -k1

This User Gave Thanks to SriniShoo For This Post:
# 4  
Old 03-26-2014
Thank you Bakunin and SriniShoo,

I have tried your solution SriniShoo and I am trying it again to be absolutely sure it is doing the right thing

Thank you
malandisa

---------- Post updated 03-26-14 at 02:36 AM ---------- Previous update was 03-25-14 at 09:15 AM ----------

I have tested this SriniShoo! thanks it works and thank you again....
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Get an output of lines in pattern 1st line then 10th line then 11th line then 20th line and so on.

Input file: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 (6 Replies)
Discussion started by: Sagar Singh
6 Replies

2. Shell Programming and Scripting

Multi line log files to single line format

I want to read the log file which was generate from other command . And the output was having multi line in log files for job name and server name. But i need to make all the logs on one line Source file 07/15/2018 17:02:00 TRANSLOG_1700 Server0005_SQL ... (2 Replies)
Discussion started by: ranjancom2000
2 Replies

3. Shell Programming and Scripting

Perl command line option '-n','-p' and multiple files: can it know a file name of a printed line?

I am looking for help in processing of those options: '-n' or '-p' I understand what they do and how to use them. But, I would like to use them with more than one file (and without any shell-loop; loading the 'perl' once.) I did try it and -n works on 2 files. Question is: - is it possible to... (6 Replies)
Discussion started by: alex_5161
6 Replies

4. UNIX for Dummies Questions & Answers

Script to delete files line by line in .txt document

Friends, I am in need to delete files from a directory on a regular basis. I want to place all the files to be deleted in delete .txt file and require a script to delete files, line by line from this delete.txt file. Please help me with this script as am new to unix scripting. (3 Replies)
Discussion started by: fop4658
3 Replies

5. Shell Programming and Scripting

How to read a two files, line by line in UNIX script and how to assign shell variable to awk ..?

Input are file and file1 file contains store.bal product.bal category.bal admin.bal file1 contains flip.store.bal ::FFFF:BADC:CD28,::FFFF:558E:11C5,6,8,2,1,::FFFF:81C8:CA8B,::FFFF:BADC:CD28,1,0,0,0,::FFFF:81C8:11C5,2,1,0,0,::FFFF:81DC:3111,1,0,1,0 store.bal.... (2 Replies)
Discussion started by: veeruasu
2 Replies

6. Shell Programming and Scripting

Computing difference based on line contents

I have the following awk script set up to copy the contents of a line that contains 0008 in each line that contains values of 1895 through 2012. awk -v OFS=" " '{val=0+substr($1,length($1)-3,4);if(val==0008){print;$1=x;y=$0}else{if(val>=1895&&val<=2012){print $1 y}else{print}}}' Output... (7 Replies)
Discussion started by: ncwxpanther
7 Replies

7. Shell Programming and Scripting

Faster Line by Line String/Date Comparison of 2 Files

Hello, I was wondering if anyone knows a faster way to search and compare strings and dates from 2 files? I'm currently using "for loop" but seems sluggish as i have to cycle through 10 directories with 10 files each containing thousands of lines. Given: -10 directories -10 files... (4 Replies)
Discussion started by: agentgrecko
4 Replies

8. Shell Programming and Scripting

how to read the contents of two files line by line and compare the line by line?

Hi All, I'm trying to figure out which are the trusted-ips and which are not using a script file.. I have a file named 'ip-list.txt' which contains some ip addresses and another file named 'trusted-ip-list.txt' which also contains some ip addresses. I want to read a line from... (4 Replies)
Discussion started by: mjavalkar
4 Replies

9. Shell Programming and Scripting

linux shell script to take variables from two different files line by line

Friends I need to have a shell script which will feed variables from two different files line-by-line. For example, I have two files - permission and file_name. Contents of permission is - 644 755 .... contents of file_name /file1 /file2 ..... Now I want 644 permission will be... (4 Replies)
Discussion started by: atanubanerji
4 Replies

10. Shell Programming and Scripting

How to calculate mean in AWK? line by line several files, thousands of lines

I'm kinda stuck on this one, I have 7 files with 30.000 lines/file like this 050 0.023 0.504336 050 0.024 0.529521 050 0.025 0.538908 050 0.026 0.537035 I want to find the mean line by line of the third column from the files named like this: Stat-f-1.dat .... Stat-f-7.dat Stat-s-1.dat... (8 Replies)
Discussion started by: AriasFco
8 Replies
Login or Register to Ask a Question