Calculating cumulative frequency using awk


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Calculating cumulative frequency using awk
# 1  
Old 12-10-2012
Calculating cumulative frequency using awk

Hi, I wanted to calculate cumulative frequency distribution of my data that involves several arithmetic calls. I did things in excel but its taking me forever. this is what I want to do:
Code:
var1.txt contains n observations which I have to compute for frequency which is given by 1/n and subsequently the cumulative frequency (cumfreq).
obs       freq (1/n)    cumfreq
2            1/7              1/7 
3            1/7              1/7+1/7
4            1/7              1/7+1/7+1/7
5            1/7              1/7+1/7+1/7+1/7
6            1/7              1/7+1/7+1/7+1/7+1/7 
7            1/7              1/7+1/7+1/7+1/7+1/7+1/7 
8            1/7              1/7+1/7+1/7+1/7+1/7+1/7+1/7

Any help on this is very much appreciated. Bunch of thanks in advance.
# 2  
Old 12-10-2012
I'm assuming you want the evaluated cumulative value and simply not the string "1/7+1/7+1/7...."

Code:
perl -lane 'if($. == 1){print; next}else{$x+=eval $F[1]; print "$F[0]\t$F[1]\t$x"}' file

# 3  
Old 12-10-2012
Hi balajesuri, thanks for your quick reply. yes, I need the evaluated cumulative frequency. There's a small glitch though, it doesn't print the first row and as such the last cumulative frequency value is a little off to 1.

And also quick question what does $. means? thanks again
# 4  
Old 12-10-2012
Quote:
Originally Posted by ida1215
Hi balajesuri, thanks for your quick reply. yes, I need the evaluated cumulative frequency. There's a small glitch though, it doesn't print the first row and as such the last cumulative frequency value is a little off to 1.
When you say first row, you mean "obs freq cumFreq", right? If so, the perl code does print this.

Quote:
Originally Posted by ida1215
And also quick question what does $. means? thanks again
In perl, scalar variables begin with a "$" symbol. That's the language grammar. Likewise, perl recognizes an array identifier by "@" as in @myArrVar a hash by "%" as in %myHash.

Last edited by balajesuri; 12-10-2012 at 02:49 PM..
# 5  
Old 12-10-2012
Hi again, Thanks much for the explanation on perl syntax. Anyway, This is the sample output am getting when I run the perl script. the columns correspond to the obs, frequency and cumulative frequency, respectively. the first value in 3rd column should be in the first row so that the last row would be equal to 1. Thanks again for looking into this.

Code:
 14.6826892861   0.0333333
 13.1351571538   0.0333333       0.0333333
 13.7802422086   0.0333333       0.0666666
 13.5688246806   0.0333333       0.0999999
 15.0765921598   0.0333333       0.1333332
 13.5727983428   0.0333333       0.1666665
 13.6297192026   0.0333333       0.1999998
 15.8496069483   0.0333333       0.2333331
 12.0589998927   0.0333333       0.2666664
 16.3124475586   0.0333333       0.2999997
 15.1092044636   0.0333333       0.333333
 16.3960654816   0.0333333       0.3666663
 14.3450835037   0.0333333       0.3999996
 14.9916008481   0.0333333       0.4333329
 15.6877187161   0.0333333       0.4666662
 13.2263422247   0.0333333       0.4999995
 16.241350749     0.0333333       0.5333328
 15.2386372531   0.0333333       0.5666661
 15.494540553     0.0333333       0.5999994
 16.0025388812   0.0333333       0.6333327
 13.8140452599   0.0333333       0.666666
 14.7094971241   0.0333333       0.6999993
 14.3596550983   0.0333333       0.7333326
 14.5479239223   0.0333333       0.7666659
 15.1549626225   0.0333333       0.7999992
 14.7984092946   0.0333333       0.8333325
 15.1006025684   0.0333333       0.8666658
 16.2515280804   0.0333333       0.8999991
 13.6803490414   0.0333333       0.9333324
 14.9813933073   0.0333333       0.9666657

# 6  
Old 12-10-2012
Ha, ha! Another classic example of changing requirements. Ok, here's some light:

if($. == 1) {print; next} ==> This part of the code checks if line in consideration is line #1 and if so prints the line as-is and proceeds to next line. This is because, from your initial sample in post #1, first line is the header line (obs freq (1/n) cumfreq).

Here's one to suit your requirement from post #5:
Code:
perl -lane '$x+=eval $F[1]; print "$F[0]\t$F[1]\t$x"' file

This User Gave Thanks to balajesuri For This Post:
# 7  
Old 12-10-2012
Thanks again and sorry bout the confusion on the header on the previous post. So, in awk its like NR=1{print}, something to that effect? Thanks again.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk or Bash: Cumulative average

For the data I would like to parse down and for each parsing I want a cumulative averaging, stored in an array that can be output. I.e. 546/NR = 546 (546+344)/NR=(546+344)/2 = etc. For N record input I want N values of the average (a block averaging effectively) Any... (3 Replies)
Discussion started by: chrisjorg
3 Replies

2. Shell Programming and Scripting

Calculating average with awk

I need to find the average from a file like: data => BW:123 M:30 RTD:0 1 0 1 0 0 1 1 1 1 0 0 1 1 0' data => BW:123 N:30 RTD:0 1 0 1 0 0 1 1 1 1 0 0 1 1 0' data => BW:123 N:30 RTD:0 1 0 1 0 0 1 1 1 1 0 0 1 1 0' data => BW:123 N:30 RTD:0 1 0 1 0 0 1 1 1 1 0 0 1 1 0' data => BW:123 N:30 RTD:0 1... (4 Replies)
Discussion started by: Slagle
4 Replies

3. Shell Programming and Scripting

Calculating the epoch time from standard time using awk and calculating the duration

Hi All, I have the following time stamp data in 2 columns Date TimeStamp(also with milliseconds) 05/23/2012 08:30:11.250 05/23/2012 08:30:15.500 05/23/2012 08:31.15.500 . . etc From this data I need the following output. 0.00( row1-row1 in seconds) 04.25( row2-row1 in... (5 Replies)
Discussion started by: ks_reddy
5 Replies

4. Shell Programming and Scripting

AWK way of calculating growth

Hi All, IS there any 'awk' way to manipulate following data? Fruit Date Count Apple 20/08/2011 5 Apple 27/08/2011 7 Apple 05/09/2011 11 Apple 12/09/2011 3 Apple 19/09/2011 25 . . . . Orange 20/08/2011 9 Orange 27/08/2011 20 Orange 27/08/2011 7 Orange 05/09/2011 15 Orange... (3 Replies)
Discussion started by: aniketdixit
3 Replies

5. Shell Programming and Scripting

Calculating frequency of values within bins

Hi, I am working with files containing 2 columns in which i need to come up with the frequency/count of values in col. 2 falling within specifics binned values of col. 1. the contents of a sample file is shown below: 15 12.5 15 11.2 16 0.2 16 1.4 17 1.6 18 4.5 17 5.6 12 8.6 11 7.2 9 ... (13 Replies)
Discussion started by: ida1215
13 Replies

6. Shell Programming and Scripting

Calculating an integer with awk

I would like to extract a number from $0 and calculate if it can be devided by 25. Though the number can also be less then 25 or bigger than 100. How do i extract the number and how can the integer be calculated? String: "all_results">39</span>I am looking for the number between "all_results"> ... (5 Replies)
Discussion started by: sdf
5 Replies

7. Shell Programming and Scripting

Help with calculating frequency of specific word in a string

Input file: #read_1 AWEAWQQRZZZQWQQWZ #read_2 ZZAQWRQTWQQQWADSADZZZ #read_3 POGZZZZZZADWRR . . Desired output file: #read_1 3 #read_1 1 #read_2 2 #read_2 3 #read_3 6 . . (3 Replies)
Discussion started by: perl_beginner
3 Replies

8. Shell Programming and Scripting

word frequency counter - awk solution?

Dear all, i need your help on this. There is a text file, i need to count word frequency for each word with frequency >40 in each line of file and output it into another file with columns like this: word1,word2,word3, ...wordn 0,0,1 1,2,0 3,2,0 etc -- each raw represents... (13 Replies)
Discussion started by: irrevocabile
13 Replies

9. Shell Programming and Scripting

Calculating cumulative frequency

Hi, I have a file containing the frequency's of an element sorted in ascending order. The file looks something like this: #Element Frequency 1 1 2 1 3 1 4 1 5 1 6 ... (5 Replies)
Discussion started by: sajal.bhatia
5 Replies

10. Shell Programming and Scripting

AWK script: decrypt text uses frequency analysis

Ez all! I have a question how to decrypt text uses letter frequency analysis. I have code which count the letters, but what i need to do after that. Can anybody help me to write a code. VERY NEEDED! My code now: #!/usr/bin/awk -f BEGIN { FS="" } { for (i=1; i <= NF; i++) { if ($i... (4 Replies)
Discussion started by: SerJel
4 Replies
Login or Register to Ask a Question