Sponsored Content
Top Forums Shell Programming and Scripting Awk based script to find the median of all individual columns in a data file Post 302653173 by ks_reddy on Friday 8th of June 2012 01:15:58 PM
Old 06-08-2012
I am getting syntax error !!

Hi binlib,

Thanks for your quick reply.
I am getting some syntax error in the code you code using gawk.
I am unable to figure it after few changes also.
Can you please try to look into it once more and correct the error.


Thanks
Sid

---------- Post updated at 10:45 PM ---------- Previous update was at 10:43 PM ----------

Hi Corona,

This perl script is not giving desired output.
I try to guess the median values how it calculated, but no clue.
What I need is for each unique value in Column1( Step) I need the median values for the remaining columns printed side by side.
I have given my required output in my post.
Can you look into your code again.

Thanks
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk based script to print the "mode(statistics term)" for each column in a data file

Hi All, Thanks all for the continued support so far. Today, I need to find the most occurring string/number(also called mode in statistics terminology) for each column in a data file (.csv type). For one column of data(1.txt) like below Sample 1 2 2 3 4 1 1 1 2 I can find the mode... (6 Replies)
Discussion started by: ks_reddy
6 Replies

2. Shell Programming and Scripting

awk based script to find the average of all the columns in a data file

Hi All, I need the modification for the below mentioned code (found in one more post https://www.unix.com/shell-programming-scripting/27161-script-generate-average-values.html) to find the average values for all the columns(but for a specific rows) and print the averages side by side. I have... (4 Replies)
Discussion started by: ks_reddy
4 Replies

3. Shell Programming and Scripting

awk based script to ignore all columns from a file which contains character strings

Hello All, I have a .CSV file where I expect all numeric data in all the columns other than column headers. But sometimes I get the files (result of statistics computation by other persons) like below( sample data) SNO,Data1,Data2,Data3 1,2,3,4 2,3,4,SOME STRING 3,4,Inf,5 4,5,4,4 I... (9 Replies)
Discussion started by: ks_reddy
9 Replies

4. Shell Programming and Scripting

Help with awk replacing identical columns based on another file

Hello, I am using Awk in UBUNTU 12.04. I have a file like following with three fields and 44706 rows. F1 A A F2 G G F3 A T I have another file like this: AL_1 F1 A A AL_2 F1 A T AL_3 F1 A A AL_1 F2 G G AL_2 F2 G A AL_3 F2 G G BO_1 F1 A A BO_2 F1 A T... (6 Replies)
Discussion started by: Homa
6 Replies

5. Shell Programming and Scripting

awk script to split file into multiple files based on many columns

So I have a space delimited file that I'd like to split into multiple files based on multiple column values. This is what my data looks like 1bc9A02 1 10 1000 FTDLNLVQALRQFLWSFRLPGEAQKIDRMMEAFAQRYCQCNNGVFQSTDTCYVLSFAIIMLNTSLHNPNVKDKPTVERFIAMNRGINDGGDLPEELLRNLYESIKNEPFKIPELEHHHHHH 1ku1A02 1 10... (9 Replies)
Discussion started by: viored
9 Replies

6. UNIX for Dummies Questions & Answers

Median calculator based on id match

I am trying to calculate the median of a column of numbers if they match an ID type on a different column. The input file has 3 columns. The column that has the ID is column 1 and the column with the values I'd like to find the median for is column 3. The file does not need to be sorted. What I... (9 Replies)
Discussion started by: verse123
9 Replies

7. Shell Programming and Scripting

Find columns in a file based on header and print to new file

Hello, I have to fish out some specific columns from a file based on the header value. I have the list of columns I need in a different file. I thought I could read in the list of headers I need, # file with header names of required columns in required order headers_file=$2 # read contents... (11 Replies)
Discussion started by: LMHmedchem
11 Replies

8. Shell Programming and Scripting

In PErl script: need to read the data one file and generate multiple files based on the data

We have the data looks like below in a log file. I want to generat files based on the string between two hash(#) symbol like below Source: #ext1#test1.tale2 drop #ext1#test11.tale21 drop #ext1#test123.tale21 drop #ext2#test1.tale21 drop #ext2#test12.tale21 drop #ext3#test11.tale21 drop... (5 Replies)
Discussion started by: Sanjeev G
5 Replies

9. Shell Programming and Scripting

awk script to find data in three file and perform replace operation

Have three files. Any other approach with regards to file concatenation or splitting, etc is appreciated If column55(billngtype) of file1 contains YMNC or YPBC then pick the value of column13(documentnumber). Now find this documentnumber in column1(Billdoc) of file2 and grep the corresponding... (4 Replies)
Discussion started by: as7951
4 Replies

10. UNIX for Advanced & Expert Users

Need Optimization shell/awk script to aggreagte (sum) for all the columns of Huge data file

Optimization shell/awk script to aggregate (sum) for all the columns of Huge data file File delimiter "|" Need to have Sum of all columns, with column number : aggregation (summation) for each column File not having the header Like below - Column 1 "Total Column 2 : "Total ... ...... (2 Replies)
Discussion started by: kartikirans
2 Replies
GPIV_ERRVEC(1)						      General Commands Manual						    GPIV_ERRVEC(1)

NAME
gpiv_errvec - Searches erroneous vectors in a PIV data file and substitutes with new values. SYNOPSIS
gpiv_errvec [-a] [-b ] [-g][-h] [-h | --help] [-i | --image_type S] [-n N] [--no_b ] [-p | --print] [-r N] [-s N] [-t F] [-u | --surround] [-v | --version] [filename] < stdin > stdout DESCRIPTION
gpiv_errvec searches erroneous outliers in a PIV data stream and substitutes with new values, if possible, i.e. if (the majority) of its surroundings are non-erroneous. Evaluation of erroneous vectors is done by looking up the Signal to Noise Ratio (SNR) value in the PIV data file or by calculating the median residu of the vector with its surroundings. A new displacement is obtained by the mean value of the surrounding values or by the location of the next higher peak in the covariance function. PIV data validation is not needed anymore while this is now done during image interrogation with gpiv_rr. Though using gpiv_errvec will not harm. The configuration parameters (containing the VALID key) may be overruled by the command line options, as explained below. Options -a Automatic threshold; calculates threshold value from residu histogram. -b Calculate statistics of the median residues and print a histogram of N bins. -g Graphical visualization of residu statistics with gnuplot (needs filename). -b is not required as this option will be set automati- cally. -h | --help On-line help. -i | --image_type S Image type or format to be used if particle displacements will have to be re-estimated at the next highest correlation peak. S may be png, tif, gif, bmp, pgm, r or gpi. Default: png. -n N Defines number of histogram of bins N for -b or -a --no_b N suppresses printing median residu histogram -p | --print Print parameters, command line options and eventually used input and output filenames to stdout. The output is identic of file- name.par, in case filename is used. -r N Erroneous vector determined by SNR of the correlation function (N = 0), median residu (N = 1) or normalized median (N = 2), i.e. the ratio between median residu with current datapoint included and between median residu of surroundigs (with datapoint excluded). -s N Substitution of the erroneous vectors by (N = 0) nothing, the local mean from the surrounding values (N = 1), median (N = 2) or next highest covariance peak (N = 3, needs filename). -t F Threshold value F of SNR or median residu to determine validity of particle displacement. -u | --surround Odd number representing the surrounding grid points to be used for local mean and median values. -v | --version Print version information on standard output, then exit successfully. filename Full name of PIV data file for input. Substitutes stdin and stdout. Output will be written to filename.erv.piv. The parameters will be written or appended to filename.par and may be used for future use by including them in ./gpivrc. If particle displacements will have to be re-estimated at the next highest correlation peak, filename is required. SEE ALSO
gpivtools AUTHOR
Gerber Van der Graaf BUGS
The program has been tested for artificial generated DPIV images. So far, no serious bugs have been found. 23 May 2005 GPIV_ERRVEC(1)
All times are GMT -4. The time now is 05:56 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy