Sponsored Content
Top Forums Shell Programming and Scripting Average, min and max in file with header, using awk Post 302771302 by kayakj on Wednesday 20th of February 2013 08:33:24 AM
Old 02-20-2013
Average, min and max in file with header, using awk

Hi,
I have a file which looks like this:
Code:
     FID        IID MISS_PHENO   N_MISS   N_GENO   F_MISS
  12AB43131   12AB43131          N    17774   906341  0.01961
  65HJ87451   65HJ87451          N    10149   906341   0.0112
  43JJ21345   43JJ21345          N     2826   906341 0.003118

I would like an awk script that extracts the average, min and max values from the last column. The following code prints the minimum value:
Code:
awk 'min=="" || $6 < min {min=$6} END{print min}' file

Putting > instead of < naturally gives me the header instead. Putting NR>1 gives me the last number of the column, here 0.003118. If I remove the header, it works fine, but I would like to skip that step. The following code, however, works fine for finding the average:
Code:
awk 'NR>1{sum+=$6}END{print "Average missing = ",sum/(NR-1)}' missing_200213.imiss

More info on file: It could be as much as 1 million lines in total. The numbers are usually between 0 and 1, ie lots of decimal numbers. After the header, the column in question is only numeric. There will always be a number, but it might be either 0 or 1.

Thank you!
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Awk search for max and min field values

hi, i have an awk script and I managed to figure out how to search the max value but Im having difficulty in searching for the min field value. BEGIN {FS=","; max=0} NF == 7 {if (max < $6) max = $6;} END { print man, min} where $6 is the column of a field separated by a comma (3 Replies)
Discussion started by: Kirichiko
3 Replies

2. Shell Programming and Scripting

get min, max and average value

hi! i have a file like the attachement. I'd like to get for each line the min, max and average values. (there is 255 values for each line) how can i get that ? i try this, is it right? BEGIN {FS = ","; OFS = ";";max=0;min=0;moy=0;total=0;freq=890} $0 !~ /Trace1:/ { ... (1 Reply)
Discussion started by: riderman
1 Replies

3. Shell Programming and Scripting

Count time min/max/average for ping

I am redirecting my ping output to a file. The sample output is like this: 64 bytes from xx.xx.xx.167: icmp_seq=4490 ttl=116 3.75 ms 2011Jul12- 15 40 16 64 bytes from xx.xx.xx.167: icmp_seq=4491 ttl=116 5.29 ms 2011Jul12- 15 40 17 64 bytes from xx.xx.xx.167: icmp_seq=4492 ttl=116 4.88 ms... (6 Replies)
Discussion started by: zorrox
6 Replies

4. Shell Programming and Scripting

Find min.max value if matching columns found using AWK

Input_ File : 2 3 4 5 1 1 0 1 2 1 -1 1 2 1 3 1 3 1 4 1 6 5 6 6 6 6 6 7 6 7 6 8 5 8 6 7 Desired output : 2 3 4 5 -1 1 4 1 6 5 6 8 5 8 6 7 (3 Replies)
Discussion started by: vasanth.vadalur
3 Replies

5. Shell Programming and Scripting

How to find the average,min,max ,total count?

Hi , Below is my sample data,I have this 8 column(A,B,C,D,E,F,G,H) in csv file. A , B ,C ,D ,E ,F,G ,H 4141,127337,24,15,20,69,72.0,-3 4141,128864,24,15,20,65,66.0,-1 4141,910053,24,15,4,4,5.0,-1 4141,910383,24,15,22,3,4.0,-1 4141,496969,24,15,14,6,-24.0,-18... (7 Replies)
Discussion started by: vinothsekark
7 Replies

6. Shell Programming and Scripting

Number of elements, average value, min & max from a list of numbers using awk

Hi all, I have a list of numbers. I need an awk command to find out the numbers of elements (number of numbers, sort to speak), the average value the min and max value. Reading the list only once, with awk. Any ideas? Thanks! (5 Replies)
Discussion started by: black_fender
5 Replies

7. Shell Programming and Scripting

Get the min avg and max with awk

aaa: 3 ms aaa: 2 ms aaa: 5 ms aaa: 10 ms .......... to get the 3 2 5 10 ...'s min avg and max something like min: 2 ms avg: 5 ms max: 10 ms (2 Replies)
Discussion started by: yanglei_fage
2 Replies

8. Shell Programming and Scripting

awk script to find min and max value

I need to find the max/min of columns 1 and 2 of a 2 column file what contains the special character ">". I know that this will find the max value of column 1. awk 'BEGIN {max = 0} {if ($1>max) max=$1} END {print max}' input.file But what if I needed to ignore special characters in the... (3 Replies)
Discussion started by: ncwxpanther
3 Replies

9. Shell Programming and Scripting

How to get min and max values using awk?

Hi, I need your kind help to get min and max values from file based on value in $5 . File1 SP12.3 stc 2240806 2240808 + ID1_N003 ID2_N003T0 SP12.3 sto 2241682 2241684 + ID1_N003 ID2_N003T0 SP12.3 XE 2239943 2240011 + ID1_N003 ID2_N003T0 SP12.3 XE 2240077 2241254 + ID1_N003 ... (12 Replies)
Discussion started by: redse171
12 Replies

10. Shell Programming and Scripting

awk Sort 2d histogram output from min(X,Y) to max(X,Y)

I've got Gnuplot-format 2D histogram data output which looks as follows. 6.5 -1.25 10.2804 6.5404 -1.25 10.4907 6.58081 -1.25 10.8087 6.62121 -1.25 10.4686 6.66162 -1.25 10.506 6.70202 -1.25 10.3084 6.74242 -1.25 9.68256 6.78283 -1.25 9.41229 6.82323 -1.25 9.43078 6.86364 -1.25 9.62408... (1 Reply)
Discussion started by: chrisjorg
1 Replies
funhist(1)							SAORD Documentation							funhist(1)

NAME
funhist - create a 1D histogram of a column (from a FITS binary table or raw event file) or an image SYNOPSIS
funhist [-n|-w|-T] <iname> [column] [[lo:hi:]bins] OPTIONS
-n # normalize bin value by the width of each bin -w # specify bin width instead of number of bins in arg3 -T # output in rdb/starbase format (tab separators) DESCRIPTION
funhist creates a one-dimensional histogram from the specified columns of a FITS Extension binary table of a FITS file (or from a non-FITS raw event file), or from a FITS image or array, and writes that histogram as an ASCII table. Alternatively, the program can perform a 1D projection of one of the image axes. The first argument to the program is required, and specifies the Funtools file: FITS table or image, raw event file, or array. If "stdin" is specified, data are read from the standard input. Use Funtools Bracket Notation to specify FITS extensions, and filters. For a table, the second argument also is required. It specifies the column to use in generating the histogram. If the data file is of type image (or array), the column is optional: if "x" (or "X"), "y" (or "Y") is specified, then a projection is performed over the x (dim1) or y (dim2) axes, respectively. (That is, this projection will give the same results as a histogram performed on a table containing the equiva- lent x,y event rows.) If no column name is specified or "xy" (or "XY") is specified for the image, then a histogram is performed on the values contained in the image pixels. The argument that follows is optional and specifies the number of bins to use in creating the histogram and, if desired, the range of bin values. For image and table histograms, the range should specify the min and max data values. For image histograms on the x and y axes, the range should specify the min and max image bin values. If this argument is omitted, the number of output bins for a table is calcu- lated either from the TLMIN/TLMAX headers values (if these exist in the table FITS header for the specified column) or by going through the data to calculate the min and max value. For an image, the number of output bins is calculated either from the DATAMIN/DATAMAX header val- ues, or by going through the data to calculate min and max value. (Note that this latter calculation might fail if the image cannot be fit in memory.) If the data are floating point (table or image) and the number of bins is not specified, an arbitrary default of 128 is used. For binary table processing, the -w (bin width) switch can be used to specify the width of each bin rather than the number of bins. Thus: funhist test.ev pha 1:100:5 means that 5 bins of width 20 are used in the histogram, while: funhist -w test.ev pha 1:100:5 means that 20 bins of width 5 are used in the histogram. The data are divvied up into the specified number of bins and the resulting 1D histogram (or projection) is output in ASCII table format. For a table, the output displays the low_edge (inclusive) and hi_edge (exclusive) values for the data. For example, a 15-row table con- taining a "pha" column whose values range from -7.5 to 7.5 can be processed thus: [sh] funhist test.ev pha # data file: /home/eric/data/test.ev # column: pha # min,max,bins: -7.5 7.5 15 bin value lo_edge hi_edge ------ --------- --------------------- --------------------- 1 22 -7.50000000 -6.50000000 2 21 -6.50000000 -5.50000000 3 20 -5.50000000 -4.50000000 4 19 -4.50000000 -3.50000000 5 18 -3.50000000 -2.50000000 6 17 -2.50000000 -1.50000000 7 16 -1.50000000 -0.50000000 8 30 -0.50000000 0.50000000 9 16 0.50000000 1.50000000 10 17 1.50000000 2.50000000 11 18 2.50000000 3.50000000 12 19 3.50000000 4.50000000 13 20 4.50000000 5.50000000 14 21 5.50000000 6.50000000 15 22 6.50000000 7.50000000 [sh] funhist test.ev pha 1:6 # data file: /home/eric/data/test.ev # column: pha # min,max,bins: 0.5 6.5 6 bin value lo_edge hi_edge ------ --------- --------------------- --------------------- 1 16 0.50000000 1.50000000 2 17 1.50000000 2.50000000 3 18 2.50000000 3.50000000 4 19 3.50000000 4.50000000 5 20 4.50000000 5.50000000 6 21 5.50000000 6.50000000 [sh] funhist test.ev pha 1:6:3 # data file: /home/eric/data/test.ev # column: pha # min,max,bins: 0.5 6.5 3 bin value lo_edge hi_edge ------ --------- --------------------- --------------------- 1 33 0.50000000 2.50000000 2 37 2.50000000 4.50000000 3 41 4.50000000 6.50000000 For a table histogram, the -n(normalize) switch can be used to normalize the bin value by the width of the bin (i.e., hi_edge-lo_edge): [sh] funhist -n test.ev pha 1:6:3 # data file: test.ev # column: pha # min,max,bins: 0.5 6.5 3 # width normalization (val/(hi_edge-lo_edge)) is applied bin value lo_edge hi_edge ------ --------------------- --------------------- --------------------- 1 16.50000000 0.50000000 2.50000000 2 6.16666667 2.50000000 4.50000000 3 4.10000000 4.50000000 6.50000000 This could used, for example, to produce a light curve with values having units of counts/second instead of counts. For an image histogram, the output displays the low and high image values (both inclusive) used to generate the histogram. For example, in the following example, 184 pixels had a value of 1, 31 had a value of 2, while only 2 had a value of 3,4,5,6, or 7: [sh] funhist test.fits # data file: /home/eric/data/test.fits # min,max,bins: 1 7 7 bin value lo_val hi_val ------ --------------------- --------------------- --------------------- 1 184.00000000 1.00000000 1.00000000 2 31.00000000 2.00000000 2.00000000 3 2.00000000 3.00000000 3.00000000 4 2.00000000 4.00000000 4.00000000 5 2.00000000 5.00000000 5.00000000 6 2.00000000 6.00000000 6.00000000 7 2.00000000 7.00000000 7.00000000 For the axis projection of an image, the output displays the low and high image bins (both inclusive) used to generate the projection. For example, in the following example, 21 counts had their X bin value of 2, etc.: [sh] funhist test.fits x 2:7 # data file: /home/eric/data/test.fits # column: X # min,max,bins: 2 7 6 bin value lo_bin hi_bin ------ --------------------- --------------------- --------------------- 1 21.00000000 2.00000000 2.00000000 2 20.00000000 3.00000000 3.00000000 3 19.00000000 4.00000000 4.00000000 4 18.00000000 5.00000000 5.00000000 5 17.00000000 6.00000000 6.00000000 6 16.00000000 7.00000000 7.00000000 [sh] funhist test.fits x 2:7:2 # data file: /home/eric/data/test.fits # column: X # min,max,bins: 2 7 2 bin value lo_bin hi_bin ------ --------------------- --------------------- --------------------- 1 60.00000000 2.00000000 4.00000000 2 51.00000000 5.00000000 7.00000000 You can use gnuplot or other plotting programs to graph the results, using a script such as: #!/bin/sh sed -e '1,/---- .*/d /^$/,$d' | awk ' BEGIN{print "set nokey; set title "funhist"; set xlabel "bin"; set ylabel "counts"; plot "-" with boxes"} {print $3, $2, $4-$3}' | gnuplot -persist - 1>/dev/null 2>&1 Similar plot commands are supplied in the script funhist.plot: funhist test.ev pha ... | funhist.plot gnuplot SEE ALSO
See funtools(7) for a list of Funtools help pages version 1.4.2 January 2, 2008 funhist(1)
All times are GMT -4. The time now is 01:59 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy