Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Standard Deviation For Every "n" lines

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Standard Deviation For Every "n" lines
# 1  
Old 01-26-2016
Standard Deviation For Every "n" lines
Hello,

I found the standard deviation work around like here

Code:
awk '{x+=$2;y+=$2^2}END{print sqrt(y/NR-(x/NR)^2)}'

My task is to calculate Std. Dev for every 40 lines in a file that has more than 11000 records.

So I was wondering on how to edit the above formula such that I can get Std. Dev for every 40 lines.

I tried so many ways by including NR%40 and it still doesn't work.

Thanks
# 2  
Old 01-26-2016
Hello jacobs.smith,

Could you please check following if that helps you, also I haven't tested it with trusting your formula. Only adding the logic to perform this logic at every 40th line of Input_file.
Code:
awk '{x+=$2;y+=$2^2;} NR%40==0{print sqrt(y/NR-(x/NR)^2)}' Input_file

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
jacobs.smith
# 3  
Old 01-27-2016
Don't forget the last records:
Code:
awk '
                {x+=$2
                 y+=$2^2  
                }
!(NR%40)        {print sqrt(y/NR-(x/NR)^2)
                 x = y = 0
                }
END             {print sqrt(y/NR-(x/NR)^2) 
                }
' Input_file

This User Gave Thanks to RudiC For This Post:
jacobs.smith
# 4  
Old 01-27-2016
What the code above does is to give you the stddev every forty lines - for ALL of the preceding records (thanks Ravinder)

Code:
rows 40 stddev x.xxx
rows 80 stddev x.xxx
......
rows 8000 stddev x.xxx

or did you mean to treat every 40 lines as a separate population of size 40, except possibly the END{} clause? - what RudiC shows except that the divisors should not be NR, rather 40 and the divisor in the END has to be calculated (maybe).

A clear description of what you want would help a lot.
Last edited by jim mcnamara; 01-27-2016 at 09:07 AM..
These 2 Users Gave Thanks to jim mcnamara For This Post:
jacobs.smith RudiC
# 5  
Old 01-27-2016
Absolutely! Missed that...

Corrected version:
Code:
awk '
                {x+=$1
                 y+=$1^2
                }
!(NR%40)        {print sqrt(y/40-(x/40)^2)
                 x = y = 0
                }
END             {print sqrt(y/(NR%40)-(x/(NR%40))^2)
                }
' file

This User Gave Thanks to RudiC For This Post:
jacobs.smith
# 6  
Old 01-27-2016
Quote:
Originally Posted by jim mcnamara
What the code above does is to give you the stddev every forty lines - for ALL of the preceding records (thanks Ravinder)

Code:
rows 40 stddev x.xxx
rows 80 stddev x.xxx
......
rows 8000 stddev x.xxx

or did you mean to treat every 40 lines as a separate population of size 40, except possibly the END{} clause? - what RudiC shows except that the divisors should not be NR, rather 40 and the divisor in the END has to be calculated (maybe).

A clear description of what you want would help a lot.
Hi Jim,

Thank you for good thoughts.

What I meant was to treat every 40rows as a set.

Lines 1 to 40 as set1
Lines 41 to 80 as set2
.......

Ravi's reply was exactly what I was looking at.

Thank you @Rudic and everyone.
# 7  
Old 01-27-2016
RavinderSingh13's proposal has

Lines 1 to 40 as set1
Lines 1 to 80 as set2
Lines 1 to 120 as set3
etc.
This User Gave Thanks to RudiC For This Post:
jacobs.smith
Login or Register to Ask a Question

Previous Thread | Next Thread

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

2. Shell Programming and Scripting

Find lines with "A" then change "E" to "X" same line

I have a bunch of random character lines like ABCEDFG. I want to find all lines with "A" and then change any "E" to "X" in the same line. ALL lines with "A" will have an "X" somewhere in it. I have tried sed awk and vi editor. I get close, not quite there. I know someone has already solved this... (10 Replies)
Discussion started by: nightwatchrenba
10 Replies

3. Shell Programming and Scripting

AWK script for standard deviation / root mean square deviation

I have a file with say 50 columns, each containing a whole lot of data. Each column contains data from a separate simulation, but each simulation is related to the data in the last (REFERENCE) column $50 I need to calculate the RMS deviation for each data line, i.e. column 1 relative to... (12 Replies)
Discussion started by: chrisjorg
12 Replies

4. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

5. Shell Programming and Scripting

Giving "read" from standard input a timeout.

I want to prompt a user for input but I want it to timeout after a specified time if no response is given. I tried the sleep command but this does not work. I am using ksh. Thanks. (10 Replies)
Discussion started by: rello
10 Replies
Login or Register to Ask a Question