Sponsored Content
Top Forums Shell Programming and Scripting Help with computing median line by line of several files. Post 302894351 by malandisa on Tuesday 25th of March 2014 03:25:14 AM
Old 03-25-2014
Help with computing median line by line of several files.

I have several files which look like this. The files are much longer than this, I have just cut out the data and there are hundreds of these files
Code:
file 1
 0.00   5.905 
 0.05   5.740 
 0.10   5.570 
 0.15   5.440 
 0.20   5.310 
 0.25   5.185 
 0.30   5.075 
 0.35   5.897 
 0.40   6.883 

file 2
 0.00   8.510 
 0.05   8.130 
 0.10   7.730 
 0.15   7.390 
 0.20   7.120 
 0.25   6.850 
 0.30   9.877 
 0.35   9.463 
 0.40   9.063 

file 3 
 0.00   5.515 
 0.05   5.375 
 0.10   5.300 
 0.15   5.200 
 0.20   4.710 
 0.25   5.280 
 0.30   5.147 
 0.35   5.055 
 0.40   4.968

I need to compute the median values of column 2 for each line... that is.. I want to get value of column 2 of each line from the different files and find the median of these values then go to the line 2 of each file and get values of column 2 and find the median and so on .... All the files have exactly equal length and the values of column 1 are the same for each file.

The output should contain column 1 values and the median value computed from the column 2 values of the different files
that is
Code:
 0.00   median of (5.905, 8.510, 5.515) 
 0.05   median of (5.740, 8.130, 5.375) 
 0.10   median of (5.570, 7.730, 5.300) 
 0.15   median of (5.440, 7.390, 5.200) 
 0.20   median of (5.310, 7.120, 4.710) 
 0.25   median of (5.185, 6.850, 5.280) 
 0.30   median of (5.075, 9.877, 5.147) 
 0.35   median of (5.897, 9.463, 5.055) 
 0.40   median of (6.883, 9.063, 4.968)

Thank you
malandisa
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to calculate mean in AWK? line by line several files, thousands of lines

I'm kinda stuck on this one, I have 7 files with 30.000 lines/file like this 050 0.023 0.504336 050 0.024 0.529521 050 0.025 0.538908 050 0.026 0.537035 I want to find the mean line by line of the third column from the files named like this: Stat-f-1.dat .... Stat-f-7.dat Stat-s-1.dat... (8 Replies)
Discussion started by: AriasFco
8 Replies

2. Shell Programming and Scripting

linux shell script to take variables from two different files line by line

Friends I need to have a shell script which will feed variables from two different files line-by-line. For example, I have two files - permission and file_name. Contents of permission is - 644 755 .... contents of file_name /file1 /file2 ..... Now I want 644 permission will be... (4 Replies)
Discussion started by: atanubanerji
4 Replies

3. Shell Programming and Scripting

how to read the contents of two files line by line and compare the line by line?

Hi All, I'm trying to figure out which are the trusted-ips and which are not using a script file.. I have a file named 'ip-list.txt' which contains some ip addresses and another file named 'trusted-ip-list.txt' which also contains some ip addresses. I want to read a line from... (4 Replies)
Discussion started by: mjavalkar
4 Replies

4. Shell Programming and Scripting

Faster Line by Line String/Date Comparison of 2 Files

Hello, I was wondering if anyone knows a faster way to search and compare strings and dates from 2 files? I'm currently using "for loop" but seems sluggish as i have to cycle through 10 directories with 10 files each containing thousands of lines. Given: -10 directories -10 files... (4 Replies)
Discussion started by: agentgrecko
4 Replies

5. Shell Programming and Scripting

Computing difference based on line contents

I have the following awk script set up to copy the contents of a line that contains 0008 in each line that contains values of 1895 through 2012. awk -v OFS=" " '{val=0+substr($1,length($1)-3,4);if(val==0008){print;$1=x;y=$0}else{if(val>=1895&&val<=2012){print $1 y}else{print}}}' Output... (7 Replies)
Discussion started by: ncwxpanther
7 Replies

6. Shell Programming and Scripting

How to read a two files, line by line in UNIX script and how to assign shell variable to awk ..?

Input are file and file1 file contains store.bal product.bal category.bal admin.bal file1 contains flip.store.bal ::FFFF:BADC:CD28,::FFFF:558E:11C5,6,8,2,1,::FFFF:81C8:CA8B,::FFFF:BADC:CD28,1,0,0,0,::FFFF:81C8:11C5,2,1,0,0,::FFFF:81DC:3111,1,0,1,0 store.bal.... (2 Replies)
Discussion started by: veeruasu
2 Replies

7. UNIX for Dummies Questions & Answers

Script to delete files line by line in .txt document

Friends, I am in need to delete files from a directory on a regular basis. I want to place all the files to be deleted in delete .txt file and require a script to delete files, line by line from this delete.txt file. Please help me with this script as am new to unix scripting. (3 Replies)
Discussion started by: fop4658
3 Replies

8. Shell Programming and Scripting

Perl command line option '-n','-p' and multiple files: can it know a file name of a printed line?

I am looking for help in processing of those options: '-n' or '-p' I understand what they do and how to use them. But, I would like to use them with more than one file (and without any shell-loop; loading the 'perl' once.) I did try it and -n works on 2 files. Question is: - is it possible to... (6 Replies)
Discussion started by: alex_5161
6 Replies

9. Shell Programming and Scripting

Multi line log files to single line format

I want to read the log file which was generate from other command . And the output was having multi line in log files for job name and server name. But i need to make all the logs on one line Source file 07/15/2018 17:02:00 TRANSLOG_1700 Server0005_SQL ... (2 Replies)
Discussion started by: ranjancom2000
2 Replies

10. UNIX for Beginners Questions & Answers

Get an output of lines in pattern 1st line then 10th line then 11th line then 20th line and so on.

Input file: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 (6 Replies)
Discussion started by: Sagar Singh
6 Replies
GPIV_ERRVEC(1)						      General Commands Manual						    GPIV_ERRVEC(1)

NAME
gpiv_errvec - Searches erroneous vectors in a PIV data file and substitutes with new values. SYNOPSIS
gpiv_errvec [-a] [-b ] [-g][-h] [-h | --help] [-i | --image_type S] [-n N] [--no_b ] [-p | --print] [-r N] [-s N] [-t F] [-u | --surround] [-v | --version] [filename] < stdin > stdout DESCRIPTION
gpiv_errvec searches erroneous outliers in a PIV data stream and substitutes with new values, if possible, i.e. if (the majority) of its surroundings are non-erroneous. Evaluation of erroneous vectors is done by looking up the Signal to Noise Ratio (SNR) value in the PIV data file or by calculating the median residu of the vector with its surroundings. A new displacement is obtained by the mean value of the surrounding values or by the location of the next higher peak in the covariance function. PIV data validation is not needed anymore while this is now done during image interrogation with gpiv_rr. Though using gpiv_errvec will not harm. The configuration parameters (containing the VALID key) may be overruled by the command line options, as explained below. Options -a Automatic threshold; calculates threshold value from residu histogram. -b Calculate statistics of the median residues and print a histogram of N bins. -g Graphical visualization of residu statistics with gnuplot (needs filename). -b is not required as this option will be set automati- cally. -h | --help On-line help. -i | --image_type S Image type or format to be used if particle displacements will have to be re-estimated at the next highest correlation peak. S may be png, tif, gif, bmp, pgm, r or gpi. Default: png. -n N Defines number of histogram of bins N for -b or -a --no_b N suppresses printing median residu histogram -p | --print Print parameters, command line options and eventually used input and output filenames to stdout. The output is identic of file- name.par, in case filename is used. -r N Erroneous vector determined by SNR of the correlation function (N = 0), median residu (N = 1) or normalized median (N = 2), i.e. the ratio between median residu with current datapoint included and between median residu of surroundigs (with datapoint excluded). -s N Substitution of the erroneous vectors by (N = 0) nothing, the local mean from the surrounding values (N = 1), median (N = 2) or next highest covariance peak (N = 3, needs filename). -t F Threshold value F of SNR or median residu to determine validity of particle displacement. -u | --surround Odd number representing the surrounding grid points to be used for local mean and median values. -v | --version Print version information on standard output, then exit successfully. filename Full name of PIV data file for input. Substitutes stdin and stdout. Output will be written to filename.erv.piv. The parameters will be written or appended to filename.par and may be used for future use by including them in ./gpivrc. If particle displacements will have to be re-estimated at the next highest correlation peak, filename is required. SEE ALSO
gpivtools AUTHOR
Gerber Van der Graaf BUGS
The program has been tested for artificial generated DPIV images. So far, no serious bugs have been found. 23 May 2005 GPIV_ERRVEC(1)
All times are GMT -4. The time now is 04:17 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy