Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Create range of bins counting values and add the percentage Post 303044451 by jiam912 on Friday 21st of February 2020 02:15:24 PM
Old 02-21-2020
Create range of bins counting values and add the percentage

Objective, create a table of range of bins to count the values for each range in column 2.

The purpose is to count at beginning all values in column 2 which contends value =0 or < 0.01
Then create the range of binnes counting values in column 2 > 0.01

The range of bins need to be extended to the MAX value in my case 20, even if these values are not found in the column.

Input file
Code:
121 0.98
121 0.98
121 1.47
121 1.47
121 0.00
121 0.00
121 0.00
121 0.10
121    2
121    3
121    7
121    0
121    0
121   10
121   12
121   15

Desired output file

Code:
            0    5    31.3%
 0.01 -  2.00    6    37.5%
 2.01 -  4.00    1     6.3%
 4.01 -  6.00    0     0.0%
 6.01 -  8.00    1     6.3%
 8.01 - 10.00    0     0.0%
10.01 - 12.00    2    12.5%
12.01 - 14.00    0     0.0%
14.01 - 16.00    1     6.3%
16.01 - 18.00    0     0.0%
18.01 - 20.00    0     0.0%
      > 20       0     0.0%
----------------------------
      Total:    16


Code Use
This code works well, but dont count values with exactly 0, and dont force the range of bins to the maximun value.

Code:
awk -v"MIN=0" -v"MAX=20" '
BEGIN   {delta = (delta=="")?2:delta
         MXBCK = (MAX / delta) + 1
        }

        {bucketNr = int(($2+delta) / delta)
         if (bucketNr > MXBCK) bucketNr = MXBCK
         cnt[bucketNr] ++
         TOT           ++ $2
         numBuckets = (numBuckets > bucketNr ? numBuckets : bucketNr)
        }

END     {
for (bucketNr=1; bucketNr<numBuckets; bucketNr++)      {end = beg + delta
         printf " %05.2f - %05.2f %6d %6.2f%%\n", beg, end, cnt[bucketNr], cnt[bucketNr] / TOT *100
         beg = end
         }
         printf "       > %5.2f %6d %6.2f%%\n", MAX, cnt[MXBCK], cnt[MXBCK] / TOT *100
         print "----------------------------"
         print "        Total:", TOT
        }
' file

Thanks in advance

Last edited by jiam912; 02-22-2020 at 06:13 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to calculate the percentage for the values in column

Hi, I am having the file which contains the following two columns. 518 _factorial 256 _main 73 _atol 52 ___do_global_ctors 170 ___main 52 ___do_g How can calculate the percentage of each value in the first column ? first need to get the sum of the first column and... (3 Replies)
Discussion started by: saleru_raja
3 Replies

2. Shell Programming and Scripting

To Create range of values

Hi, I have a file with the below like values with integers only in sorted order (May or may not be in sequence) Eg: File1.txt ----------- 1 2 3 4 5 6 . . . . . 10000 My requirement here is to create a range of values out put to a temp k (4 Replies)
Discussion started by: shiva447
4 Replies

3. Shell Programming and Scripting

Howto: easy date range iteration/counting on GNU systems in the shell

Should work in any shell, but requires GNU date, although GNU date seems only to be happy for input dates between 1902 and 2037, inclusive (49673 days). Assume $a and $b hold two dates, e.g. set a=2010-03-27 set b=2010-04-04 Marginally faster: iterator: seq -f "$a +%1.0f days" 1 50000 |... (0 Replies)
Discussion started by: laddiebuck
0 Replies

4. UNIX for Dummies Questions & Answers

Working out the percentage between two values

Hi there, I am totally new to Unix, I am trying to work out the percentage between two values in a ksh shell script and assign the result to a variable. Value1=577 Values2=244 So the calculation would be as follows: ((Value1 - Value2) / Value1) * 100 How would I be able to achieve... (2 Replies)
Discussion started by: NextLevelAndi
2 Replies

5. Shell Programming and Scripting

return counting per hour percentage

We have a monitoring process for a load in unix box, during this process we are writing logs statements for each record, and during this process we are showing the counts per hour. Here is that how we are following log files statements: (just two lines printed here), these statements logged at... (1 Reply)
Discussion started by: skkuchipudi
1 Replies

6. Shell Programming and Scripting

Calculating frequency of values within bins

Hi, I am working with files containing 2 columns in which i need to come up with the frequency/count of values in col. 2 falling within specifics binned values of col. 1. the contents of a sample file is shown below: 15 12.5 15 11.2 16 0.2 16 1.4 17 1.6 18 4.5 17 5.6 12 8.6 11 7.2 9 ... (13 Replies)
Discussion started by: ida1215
13 Replies

7. Shell Programming and Scripting

Convert Column Values to a Range of Values

I have a list of columns with values that I need to transform into a row containing the range of each column. For example: "Column A" 1 2 3 4 10 12 14 15 16 17 18 "Column B" 1 4 5 6 (4 Replies)
Discussion started by: newbio
4 Replies

8. Shell Programming and Scripting

Delete row if both percentage values are equal to zero

Hello, I have compiled a script but I have stucked at one point. Each line contains two pcs of % value and what I want to do is to delete any line if both % values are zero. data: expected output: ow3 should be deleted as both percentage value in related line are equal to zero. ... (2 Replies)
Discussion started by: baris35
2 Replies

9. Shell Programming and Scripting

Create range of values and count values.

Gents, It is possible to generate a range of values according to column 1 and count the total of rows in the range. example input 15.3 15.5 15.8 15.9 16.0 16.1 16.8 17.0 17.5 18.0 output desired 15.0 - 15.9 = 4 (10 Replies)
Discussion started by: jiam912
10 Replies

10. UNIX for Beginners Questions & Answers

Create bins with totals and percentage

I would like to create bins to get histogram with totals and percentage, e.g. starting from 0. If possible to set the minimum and maximum value in the bins ( in my case value min=0 and max=20 ) Input file 8 5 10 1 11 4 12 4 12 4 13 5 16 7 18 9 16 9 17 7 18 5 19 5 20 1 21 7 (10 Replies)
Discussion started by: jiam912
10 Replies
funhist(1)							SAORD Documentation							funhist(1)

NAME
funhist - create a 1D histogram of a column (from a FITS binary table or raw event file) or an image SYNOPSIS
funhist [-n|-w|-T] <iname> [column] [[lo:hi:]bins] OPTIONS
-n # normalize bin value by the width of each bin -w # specify bin width instead of number of bins in arg3 -T # output in rdb/starbase format (tab separators) DESCRIPTION
funhist creates a one-dimensional histogram from the specified columns of a FITS Extension binary table of a FITS file (or from a non-FITS raw event file), or from a FITS image or array, and writes that histogram as an ASCII table. Alternatively, the program can perform a 1D projection of one of the image axes. The first argument to the program is required, and specifies the Funtools file: FITS table or image, raw event file, or array. If "stdin" is specified, data are read from the standard input. Use Funtools Bracket Notation to specify FITS extensions, and filters. For a table, the second argument also is required. It specifies the column to use in generating the histogram. If the data file is of type image (or array), the column is optional: if "x" (or "X"), "y" (or "Y") is specified, then a projection is performed over the x (dim1) or y (dim2) axes, respectively. (That is, this projection will give the same results as a histogram performed on a table containing the equiva- lent x,y event rows.) If no column name is specified or "xy" (or "XY") is specified for the image, then a histogram is performed on the values contained in the image pixels. The argument that follows is optional and specifies the number of bins to use in creating the histogram and, if desired, the range of bin values. For image and table histograms, the range should specify the min and max data values. For image histograms on the x and y axes, the range should specify the min and max image bin values. If this argument is omitted, the number of output bins for a table is calcu- lated either from the TLMIN/TLMAX headers values (if these exist in the table FITS header for the specified column) or by going through the data to calculate the min and max value. For an image, the number of output bins is calculated either from the DATAMIN/DATAMAX header val- ues, or by going through the data to calculate min and max value. (Note that this latter calculation might fail if the image cannot be fit in memory.) If the data are floating point (table or image) and the number of bins is not specified, an arbitrary default of 128 is used. For binary table processing, the -w (bin width) switch can be used to specify the width of each bin rather than the number of bins. Thus: funhist test.ev pha 1:100:5 means that 5 bins of width 20 are used in the histogram, while: funhist -w test.ev pha 1:100:5 means that 20 bins of width 5 are used in the histogram. The data are divvied up into the specified number of bins and the resulting 1D histogram (or projection) is output in ASCII table format. For a table, the output displays the low_edge (inclusive) and hi_edge (exclusive) values for the data. For example, a 15-row table con- taining a "pha" column whose values range from -7.5 to 7.5 can be processed thus: [sh] funhist test.ev pha # data file: /home/eric/data/test.ev # column: pha # min,max,bins: -7.5 7.5 15 bin value lo_edge hi_edge ------ --------- --------------------- --------------------- 1 22 -7.50000000 -6.50000000 2 21 -6.50000000 -5.50000000 3 20 -5.50000000 -4.50000000 4 19 -4.50000000 -3.50000000 5 18 -3.50000000 -2.50000000 6 17 -2.50000000 -1.50000000 7 16 -1.50000000 -0.50000000 8 30 -0.50000000 0.50000000 9 16 0.50000000 1.50000000 10 17 1.50000000 2.50000000 11 18 2.50000000 3.50000000 12 19 3.50000000 4.50000000 13 20 4.50000000 5.50000000 14 21 5.50000000 6.50000000 15 22 6.50000000 7.50000000 [sh] funhist test.ev pha 1:6 # data file: /home/eric/data/test.ev # column: pha # min,max,bins: 0.5 6.5 6 bin value lo_edge hi_edge ------ --------- --------------------- --------------------- 1 16 0.50000000 1.50000000 2 17 1.50000000 2.50000000 3 18 2.50000000 3.50000000 4 19 3.50000000 4.50000000 5 20 4.50000000 5.50000000 6 21 5.50000000 6.50000000 [sh] funhist test.ev pha 1:6:3 # data file: /home/eric/data/test.ev # column: pha # min,max,bins: 0.5 6.5 3 bin value lo_edge hi_edge ------ --------- --------------------- --------------------- 1 33 0.50000000 2.50000000 2 37 2.50000000 4.50000000 3 41 4.50000000 6.50000000 For a table histogram, the -n(normalize) switch can be used to normalize the bin value by the width of the bin (i.e., hi_edge-lo_edge): [sh] funhist -n test.ev pha 1:6:3 # data file: test.ev # column: pha # min,max,bins: 0.5 6.5 3 # width normalization (val/(hi_edge-lo_edge)) is applied bin value lo_edge hi_edge ------ --------------------- --------------------- --------------------- 1 16.50000000 0.50000000 2.50000000 2 6.16666667 2.50000000 4.50000000 3 4.10000000 4.50000000 6.50000000 This could used, for example, to produce a light curve with values having units of counts/second instead of counts. For an image histogram, the output displays the low and high image values (both inclusive) used to generate the histogram. For example, in the following example, 184 pixels had a value of 1, 31 had a value of 2, while only 2 had a value of 3,4,5,6, or 7: [sh] funhist test.fits # data file: /home/eric/data/test.fits # min,max,bins: 1 7 7 bin value lo_val hi_val ------ --------------------- --------------------- --------------------- 1 184.00000000 1.00000000 1.00000000 2 31.00000000 2.00000000 2.00000000 3 2.00000000 3.00000000 3.00000000 4 2.00000000 4.00000000 4.00000000 5 2.00000000 5.00000000 5.00000000 6 2.00000000 6.00000000 6.00000000 7 2.00000000 7.00000000 7.00000000 For the axis projection of an image, the output displays the low and high image bins (both inclusive) used to generate the projection. For example, in the following example, 21 counts had their X bin value of 2, etc.: [sh] funhist test.fits x 2:7 # data file: /home/eric/data/test.fits # column: X # min,max,bins: 2 7 6 bin value lo_bin hi_bin ------ --------------------- --------------------- --------------------- 1 21.00000000 2.00000000 2.00000000 2 20.00000000 3.00000000 3.00000000 3 19.00000000 4.00000000 4.00000000 4 18.00000000 5.00000000 5.00000000 5 17.00000000 6.00000000 6.00000000 6 16.00000000 7.00000000 7.00000000 [sh] funhist test.fits x 2:7:2 # data file: /home/eric/data/test.fits # column: X # min,max,bins: 2 7 2 bin value lo_bin hi_bin ------ --------------------- --------------------- --------------------- 1 60.00000000 2.00000000 4.00000000 2 51.00000000 5.00000000 7.00000000 You can use gnuplot or other plotting programs to graph the results, using a script such as: #!/bin/sh sed -e '1,/---- .*/d /^$/,$d' | awk ' BEGIN{print "set nokey; set title "funhist"; set xlabel "bin"; set ylabel "counts"; plot "-" with boxes"} {print $3, $2, $4-$3}' | gnuplot -persist - 1>/dev/null 2>&1 Similar plot commands are supplied in the script funhist.plot: funhist test.ev pha ... | funhist.plot gnuplot SEE ALSO
See funtools(7) for a list of Funtools help pages version 1.4.2 January 2, 2008 funhist(1)
All times are GMT -4. The time now is 06:18 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy