Sponsored Content
Top Forums Shell Programming and Scripting Average calculation based on number of rows Post 302428622 by Gery on Thursday 10th of June 2010 10:52:53 AM
Old 06-10-2010
Yes, the first two value blocks:

Code:
272134.548 6680572.715
272134.545 6680572.711
272134.546 6680572.713
272134.548 6680572.706
272134.545 6680572.721
272134.543 6680572.710
272134.544 6680572.715
272134.543 6680572.705
272134.540 6680572.720
272134.544 6680572.721

272134.539 6680572.714
272134.260 6680573.092

I need to calculate the average of the first column and get one value, so the sum is:
Code:
 
272134.548
272134.545
272134.546
272134.548
272134.545
272134.543
272134.544
272134.543
272134.540
272134.544

= 272134.5446 (2721345.446/10)

The same with the second column, and get one value.

But the point is that I have to do this for each value block (where both blocks are separated by a blank line). I have 20 000 rows, like 500 value blocks. So, for the two blocks showed above, I will get, let's say:

272134.543 6680572.705 (first value block)

272134.546 6680572.713 (second value block)

It is not important how the pair of output values is (ie. comma-separated, etc), it is just important the pair of output values.

Thanks for your answer.

Last edited by vgersh99; 06-10-2010 at 01:28 PM.. Reason: code tags, please!
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Average completion time calculation?

I've been trying all night to come up with a script that will take a file that contains job completion times like this as input: 18:30 17:45 16:39 18:01 17:50 ... and figure the Average completion time. I've tried several things, and I just can't seem to get it to figure correctly. I'm... (5 Replies)
Discussion started by: Seawall
5 Replies

2. Shell Programming and Scripting

Splitting file based on number of rows

Hi, I'm, new to shell scripting, I have a requirement where I have to split an incoming file into separate files each containing a maximum of 3 million rows. For e.g: if my incoming file say In.txt has 8 mn rows then I need to create 3 files, in which two will 3 mn rows and one will contain 2... (2 Replies)
Discussion started by: wahi80
2 Replies

3. UNIX for Dummies Questions & Answers

Calculating the Number of Rows and Average

Hi All I like to know how can we calculate the number of rows and the average of the values present in the file. I will not know what will be the rowcount, which will be dynamic in nature of the file. eg. 29 33 48 30 28 (6 Replies)
Discussion started by: pk_eee
6 Replies

4. Shell Programming and Scripting

Script to find the average of a given column and also for specified number of rows??

Hi friends I have 100 files in my directory. Each file look like this.. Temp1 Temp2 Temp3 MAS 1 2 3 MAS 4 5 6 MAS 7 8 9 Delhi 10 11 12 Delhi 13 14 15 Delhi 16 17 ... (4 Replies)
Discussion started by: ks_reddy
4 Replies

5. Shell Programming and Scripting

Script to find the average of a given column and also for specified number of rows?

Hi Friends, In continuation to my earlier post https://www.unix.com/shell-programming-scripting/99166-script-find-average-given-column-also-specified-number-rows.html I am extending my problem as follows. Input: Column1 Column2 MAS 1 MAS 4 ... (2 Replies)
Discussion started by: ks_reddy
2 Replies

6. UNIX for Dummies Questions & Answers

count number of rows based on other column values

Could anybody help with this? I have input below ..... david,39 david,39 emelie,40 clarissa,22 bob,42 bob,42 tim,32 bob,39 david,38 emelie,47 what i want to do is count how many names there are with different ages, so output would be like this .... david,2 emelie,2 clarissa,1... (3 Replies)
Discussion started by: itsme999
3 Replies

7. UNIX for Dummies Questions & Answers

Sum the rows number based on first field string value

Hi, I have a file like this one h1 4.70650E-04 4.70650E-04 4.70650E-04 h2 1.92912E-04 1.92912E-04 1.92912E-04 h3A 3.10160E-11 2.94562E-11 2.78458E-11 h4 0.00000E+00 0.00000E+00 0.00000E+00 h1 1.18164E-12 2.74150E-12 4.35187E-12 h1 7.60813E-01 7.60813E-01 7.60813E-01... (5 Replies)
Discussion started by: f_o_555
5 Replies

8. Shell Programming and Scripting

Split File based on number of rows

Hi I have a requirement, where i will receive multiple files in a folder (say: /fol1/fol2/). There will be at least 14 to 16 files. The size of the files will different, some may be 80GB or 90GB, some may be less than 5 GB (and the size of the files are very unpredictable). But the names of the... (10 Replies)
Discussion started by: kpk_ds
10 Replies

9. Shell Programming and Scripting

Average select rows

I have no idea how to even get started with this script. I need to average field 3 for each of the unique identifiers found in field 1. However, I only want to average these rows when field 2 is equal to 1506 - 2000 (note that i replaced the values field 2 for security reasons, but the real... (6 Replies)
Discussion started by: ncwxpanther
6 Replies

10. UNIX for Beginners Questions & Answers

Help with average calculation.

i have a file with 2 columns. i want to calculate the average of column 1 based on the values of column 2. here's how the file looks like. i want to calculate the sums of numbers corresponding to 1 and then calculate the average. same for numbers corresponding to zero. any help with a code would... (1 Reply)
Discussion started by: onerokeyz
1 Replies
fundisp(1)							SAORD Documentation							fundisp(1)

NAME
fundisp - display data in a Funtools data file SYNOPSIS
fundisp [-f format] [-l] [-n] [-T] <iname> [columns|bitpix=n] OPTIONS
-f # format string for display -l # display image as a list containing the columns X, Y, VAL -n # don't output header -F [c] # use specified character as column separator (def: space) -T # output in rdb/starbase format (tab separators) DESCRIPTION
fundisp displays the data in the specified FITS Extension and/or Image Section of a FITS file, or in a Section of a non-FITS array or raw event file. The first argument to the program specifies the FITS input image, array, or raw event file to display. If "stdin" is specified, data are read from the standard input. Use Funtools Bracket Notation to specify FITS extensions, image sections, and filters. If the data being displayed are columns (either in a FITS binary table or a raw event file), the individual rows are listed. Filters can be added using bracket notation. Thus: [sh] fundisp "test.ev[time-(int)time>.15]" X Y PHA PI TIME DX DY ------- ------- ------- --------- ---------------- ---------- ---------- 10 8 10 8 17.1600 8.50 10.50 9 9 9 9 17.1600 9.50 9.50 10 9 10 9 18.1600 9.50 10.50 10 9 10 9 18.1700 9.50 10.50 8 10 8 10 17.1600 10.50 8.50 9 10 9 10 18.1600 10.50 9.50 9 10 9 10 18.1700 10.50 9.50 10 10 10 10 19.1600 10.50 10.50 10 10 10 10 19.1700 10.50 10.50 10 10 10 10 19.1800 10.50 10.50 [NB: The FITS binary table test file test.ev, as well as the FITS image test.fits, are contained in the funtools funtest directory.] When a table is being displayed using fundisp, a second optional argument can be used to specify the columns to display. For example: [sh] fundisp "test.ev[time-(int)time>=.99]" "x y time" X Y TIME -------- -------- --------------------- 5 -6 40.99000000 4 -5 59.99000000 -1 0 154.99000000 -2 1 168.99000000 -3 2 183.99000000 -4 3 199.99000000 -5 4 216.99000000 -6 5 234.99000000 -7 6 253.99000000 The special column $REGION can be specified to display the region id of each row: [sh $] fundisp "test.ev[time-(int)time>=.99&&annulus(0 0 0 10 n=3)]" 'x y time $REGION' X Y TIME REGION -------- -------- --------------------- ---------- 5 -6 40.99000000 3 4 -5 59.99000000 2 -1 0 154.99000000 1 -2 1 168.99000000 1 -3 2 183.99000000 2 -4 3 199.99000000 2 -5 4 216.99000000 2 -6 5 234.99000000 3 -7 6 253.99000000 3 Here only rows with the proper fractional time and whose position also is within one of the three annuli are displayed. Columns can be excluded from display using a minus sign before the column: [sh $] fundisp "test.ev[time-(int)time>=.99]" "-time" X Y PHA PI DX DY -------- -------- -------- ---------- ----------- ----------- 5 -6 5 -6 5.50 -6.50 4 -5 4 -5 4.50 -5.50 -1 0 -1 0 -1.50 0.50 -2 1 -2 1 -2.50 1.50 -3 2 -3 2 -3.50 2.50 -4 3 -4 3 -4.50 3.50 -5 4 -5 4 -5.50 4.50 -6 5 -6 5 -6.50 5.50 -7 6 -7 6 -7.50 6.50 All columns except the time column are displayed. The special column $N can be specified to display the ordinal value of each row. Thus, continuing the previous example: fundisp "test.ev[time-(int)time>=.99]" '-time $n' X Y PHA PI DX DY N ------- -------- -------- ---------- ----------- ----------- ---------- 5 -6 5 -6 5.50 -6.50 337 4 -5 4 -5 4.50 -5.50 356 -1 0 -1 0 -1.50 0.50 451 -2 1 -2 1 -2.50 1.50 465 -3 2 -3 2 -3.50 2.50 480 -4 3 -4 3 -4.50 3.50 496 -5 4 -5 4 -5.50 4.50 513 -6 5 -6 5 -6.50 5.50 531 -7 6 -7 6 -7.50 6.50 550 Note that the column specification is enclosed in single quotes to protect '$n' from begin expanded by the shell. In general, the rules for activating and de-activating columns are: o If only exclude columns are specified, then all columns but the exclude columns will be activated. o If only include columns are specified, then only the specified columns are activated. o If a mixture of include and exclude columns are specified, then all but the exclude columns will be active; this last case is ambiguous and the rule is arbitrary. In addition to specifying columns names explicitly, the special symbols + and - can be used to activate and de-activate all columns. This is useful if you want to activate the $REGION column along with all other columns. According to the rules, the syntax "$REGION" only acti- vates the region column and de-activates the rest. Use "+ $REGION" to activate all columns as well as the region column. If the data being displayed are image data (either in a FITS primary image, a FITS image extension, or an array file), an mxn pixel display is produced, where m and n are the dimensions of the image. By default, pixel values are displayed using the same data type as in the file. However, for integer data where the BSCALE and BZERO header parameters are present, the data is displayed as floats. In either case, the display data type can be overridden using an optional second argument of the form: bitpix=n where n is 8,16,32,-32,-64, for unsigned char, short, int, float and double, respectively. Of course, running fundisp on anything but the smallest image usually results in a display whose size makes it unreadable. Therefore, one can uses bracket notation (see below) to apply section and/or blocking to the image before generating a display. For example: [sh] fundisp "test.fits[2:6,2:7]" bitpix=-32 2 3 4 5 6 ---------- ---------- ---------- ---------- ---------- 2: 3.00 4.00 5.00 6.00 7.00 3: 4.00 5.00 6.00 7.00 8.00 4: 5.00 6.00 7.00 8.00 9.00 5: 6.00 7.00 8.00 9.00 10.00 6: 7.00 8.00 9.00 10.00 11.00 7: 8.00 9.00 10.00 11.00 12.00 Note that is is possible to display a FITS binary table as an image simply by passing the table through funimage first: [sh] ./funimage test.ev stdout | fundisp "stdin[2:6,2:7]" bitpix=8 2 3 4 5 6 ------- ------- ------- ------- ------- 2: 3 4 5 6 7 3: 4 5 6 7 8 4: 5 6 7 8 9 5: 6 7 8 9 10 6: 7 8 9 10 11 7: 8 9 10 11 12 If the -l (list) switch is used, then an image is displayed as a list containing the columns: X, Y, VAL. For example: fundisp -l "test1.fits[2:6,2:7]" bitpix=-32 X Y VAL ---------- ---------- ----------- 2 2 6.00 3 2 1.00 4 2 1.00 5 2 1.00 6 2 1.00 2 3 1.00 3 3 5.00 4 3 1.00 5 3 1.00 6 3 1.00 2 4 1.00 3 4 1.00 4 4 4.00 5 4 1.00 6 4 1.00 2 5 1.00 3 5 1.00 4 5 1.00 5 5 3.00 6 5 1.00 2 6 1.00 3 6 1.00 4 6 1.00 5 6 1.00 6 6 2.00 2 7 1.00 3 7 1.00 4 7 1.00 5 7 1.00 6 7 1.00 If the -n (nohead) switch is used, then no header is output for tables. This is useful, for example, when fundisp output is being directed into gnuplot. The fundisp program uses a default set of display formats: datatype TFORM format -------- ----- -------- double D "%21.8f" float E "%11.2f" int J "%10d" short I "%8d" byte B "%6d" string A "%12.12s" bits X "%8x" logical L "%1x" Thus, the default display of 1 double and 2 shorts gives: [sh] fundisp snr.ev "time x y" TIME X Y --------------------- -------- -------- 79494546.56818075 546 201 79488769.94469175 548 201 ... You can change the display format for individual columns or for all columns of a given data types by means of the -f switch. The format string that accompanies -f is a space-delimited list of keyword=format values. The keyword values can either be column names (in which case the associated format pertains only to that column) or FITS table TFORM specifiers (in which case the format pertains to all columns having that data type). For example, you can change the double and short formats for all columns like this: [sh] fundisp -f "D=%22.11f I=%3d" snr.ev "time x y" TIME X Y ---------------------- --- --- 79494546.56818075478 546 201 79488769.94469174743 548 201 ... Alternatively, you can change the format of the time and x columns like this: [sh] fundisp -f "time=%22.11f x=%3d" snr.ev "time x y" TIME X Y ---------------------- --- -------- 79494546.56818075478 546 201 79488769.94469174743 548 201 ... Note that there is a potential conflict if a column has the same name as one of the TFORM specifiers. In the examples above, the the "X" column in the table has the same name as the X (bit) datatype. To resolve this conflict, the format string is processed such that TFORM datatype specifiers are checked for first, using a case-sensitive comparison. If the specified format value is not an upper case TFORM value, then a case-insensitive check is made on the column name. This means that, in the examples above, "X=%3d" will refer to the X (bit) datatype, while "x=%3d" will refer to the X column: [sh] fundisp -f "X=%3d" snr.ev "x y" X Y -------- -------- 546 201 548 201 ... [sh] fundisp -f "x=%3d" snr.ev "x y" X Y --- -------- 546 201 548 201 ... As a rule, therefore, it is best always to specify the column name in lower case and TFORM data types in upper case. The -f [format] will change the format for a single execution of fundisp. You also can use the FUN_FORMAT envronment variable to change the format for all invocations of fundisp. The format of this environment variable's value is identical to that used with the -f switch. This global value can be overridden in individual cases by use of the -f [format] switch. Caveats: Please also note that it is the user's responsibility to match the format specifier to the column data type correctly. Also note that, in order to maintain visual alignment between names and columns, the column name will be truncated (on the left) if the format width is less than the length of the name. However, truncation is not performed if the output is in RDB format (using the -T switch). [An older-style format string is supported but deprecated. It consists of space-delimited C format statements for all data types, specified in the following order: double float int short byte string bit. This order of the list is based on the assumption that people generally will want to change the float formats. If "-" is entered instead of a format statement for a given data type, the default format is used. Also, the format string can be termi- nated without specifying all formats, and defaults will be used for the rest of the list. Note that you must supply a minimum field width, i.e., "%6d" and "%-6d" are legal, "%d" is not legal. By using -f [format], you can change the double and short formats like this: [sh] fundisp -f "22.11f - - 3d" snr.ev "time x y" TIME X Y ---------------------- --- --- 79494546.56818075478 546 201 79488769.94469174743 548 201 ... NB: This format is deprecated and will be removed in a future release.] The -F[c] switch can be used to specify a (single-character) column separator (where the default is a space). Note that column formatting will almost certainly also add spaces to pad individual columns to the required width. These can be removed with a program such as sed, at the cost of generating unaligned columns. For example: fundisp -F',' snr.ev'[cir 512 512 .1]' X, Y, PHA, PI, TIME, DX, DY 512, 512, 6, 7, 79493997.45854475, 578, 574 512, 512, 8, 9, 79494575.58943175, 579, 573 512, 512, 5, 6, 79493631.03866175, 578, 575 512, 512, 5, 5, 79493290.86521725, 578, 575 512, 512, 8, 9, 79493432.00990875, 579, 573 fundisp -F',' snr.ev'[cir 512 512 .1]' | sed 's/ *, */,/g' X,Y,PHA,PI,TIME,DX,DY 512,512,6,7,79493997.45854475,578,574 512,512,8,9,79494575.58943175,579,573 512,512,5,6,79493631.03866175,578,575 512,512,5,5,79493290.86521725,578,575 512,512,8,9,79493432.00990875,579,573 fundisp -f "x=%3d y=%3d pi=%1d pha=%1d time=%20.11f dx=%3d dy=%3d" -F',' snr.ev'[cir 512 512 .1]' | sed 's/ *, */,/g' X,Y,A,I,TIME,DX,DY ---,---,-,-,--------------------,---,--- 512,512,6,7,79493997.45854474604,578,574 512,512,8,9,79494575.58943174779,579,573 512,512,5,6,79493631.03866174817,578,575 512,512,5,5,79493290.86521725357,578,575 512,512,8,9,79493432.00990875065,579,573 If the -T (rdb table) switch is used, the output will conform to starbase/rdb data base format: tabs will be inserted between columns rather than spaces. This format is not available when displaying image pixels (except in conjunction with the -l switch). Finally, note that fundisp can be used to create column filters from the auxiliary tables in a FITS file. For example, the following shell code will generate a good-time interval (GTI) filter for X-ray data files that contain a standard GTI extension: #!/bin/sh sed '1,/---- .*/d /^$/,$d' | awk 'tot>0{printf "||"};{printf "time="$1":"$2; tot++}' If this script is placed in a file called "mkgti", it can be used in a command such as: fundisp foo.fits"[GTI]" | mkgti > gti.filter The resulting filter file can then be used in various funtools programs: funcnts foo.fits"[@gti.filter]" ... to process only the events in the good-time intervals. SEE ALSO
See funtools(7) for a list of Funtools help pages version 1.4.2 January 2, 2008 fundisp(1)
All times are GMT -4. The time now is 03:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy