Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Calculating Running Variance Using Awk
Top Forums Shell Programming and Scripting Calculating Running Variance Using Awk Post 302665979 by Jahn on Tuesday 3rd of July 2012 01:11:51 PM
Old 07-03-2012
Calculating Running Variance Using Awk
Hi all,

I am attempting to calculate a running variance for a file containing a column of numbers. I am using the formula variance=sum((x-mean(x))^2)/(n-1), where x is the value on the current row, and mean(x) is the average of all of the values up until that row. n represents the total number of rows up until the current row.

For example, given a column of three numbers:

Code:
100
100
-50

The variance should be:

Code:
0
0
7500

Because when we get to row three, mean(x) = (100+100+(-50))/3 = 50, and the variance would therefore be:

Code:
variance= ((100-50)^2 + (100-50)^2 + (-50-50)^2)/(3-1) = (50^2 + 50^2 + 100^2) / 2 = 15000/2 = 7500

My question is, how do I do this with awk to generate a running total of the variance per line? I am using awk to perform several other mathematical operations on my data, so I would prefer to use it for this operation as well; however, if there is a more appropriate tool for doing this, I would like to hear about it.


Thanks,

-Jahn
Last edited by Scott; 07-03-2012 at 02:28 PM.. Reason: Added code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

calculating variance in perl programming

#!/usr/bin/perl -w use strict; open(FH,"$ARGV") or die; my @temp=<FH>; close FH; my $mean = Mean(\@temp); my $var = variance(\@temp); print "$var\n"; sub estimate_variance { my ($arrayref) = @_; my ($mean,$result) = (mean($arrayref),0); foreach (@$arrayref) {... (4 Replies)
Discussion started by: cdfd123
4 Replies

2. Shell Programming and Scripting

Calculating totals in AWK

Hello, With the following small script I list the size of documents belonging to a certain user by each time selecting the bytes-field of that file ($7). Now it fills the array with every file it finds so in the end the output of some users contains up to 200.000 numbers. So how can I calculate... (7 Replies)
Discussion started by: Hille
7 Replies

3. Shell Programming and Scripting

Awk total and variance

File1 0358 Not Visible ***:* NA:NA RDF1+TDEV Grp'd (M) RW 102413 0359 Not Visible ***:* NA:NA RDF1+TDEV N/Grp'd (m) RW - 035A Not Visible ***:* NA:NA RDF1+TDEV N/Grp'd (m) RW - 035B Not Visible ***:* NA:NA ... (2 Replies)
Discussion started by: greycells
2 Replies

4. Shell Programming and Scripting

Calculating an integer with awk

I would like to extract a number from $0 and calculate if it can be devided by 25. Though the number can also be less then 25 or bigger than 100. How do i extract the number and how can the integer be calculated? String: "all_results">39</span>I am looking for the number between "all_results"> ... (5 Replies)
Discussion started by: sdf
5 Replies

5. Shell Programming and Scripting

AWK way of calculating growth

Hi All, IS there any 'awk' way to manipulate following data? Fruit Date Count Apple 20/08/2011 5 Apple 27/08/2011 7 Apple 05/09/2011 11 Apple 12/09/2011 3 Apple 19/09/2011 25 . . . . Orange 20/08/2011 9 Orange 27/08/2011 20 Orange 27/08/2011 7 Orange 05/09/2011 15 Orange... (3 Replies)
Discussion started by: aniketdixit
3 Replies

6. Shell Programming and Scripting

Calculating the epoch time from standard time using awk and calculating the duration

Hi All, I have the following time stamp data in 2 columns Date TimeStamp(also with milliseconds) 05/23/2012 08:30:11.250 05/23/2012 08:30:15.500 05/23/2012 08:31.15.500 . . etc From this data I need the following output. 0.00( row1-row1 in seconds) 04.25( row2-row1 in... (5 Replies)
Discussion started by: ks_reddy
5 Replies

7. Shell Programming and Scripting

Calculating average with awk

I need to find the average from a file like: data => BW:123 M:30 RTD:0 1 0 1 0 0 1 1 1 1 0 0 1 1 0' data => BW:123 N:30 RTD:0 1 0 1 0 0 1 1 1 1 0 0 1 1 0' data => BW:123 N:30 RTD:0 1 0 1 0 0 1 1 1 1 0 0 1 1 0' data => BW:123 N:30 RTD:0 1 0 1 0 0 1 1 1 1 0 0 1 1 0' data => BW:123 N:30 RTD:0 1... (4 Replies)
Discussion started by: Slagle
4 Replies

8. Shell Programming and Scripting

AWK sample variance

I would like to calculate 1/n In awk, I wrote the following line for the sigma summation: { summ+=($1-average)^2 } Full code: BEGIN { Print "This script calculate error estimates"; sum=0 } { sum+=$1; n++ } END { average = sum/n } BEGIN { summ=0 } { summ+=($1-average)^2 } END { print... (8 Replies)
Discussion started by: chrisjorg
8 Replies

9. Shell Programming and Scripting

Calculating the running time

Hi All, I want to run a utility for all the process id that are running for more than 15 mins. I have captured process id's and the time that they were run in a file like below 1st column represnts the process ids and the 2nd one is the Time < 21014 01:00 21099 01:00 24361 01:03 24406... (5 Replies)
Discussion started by: r_t_1601
5 Replies

10. UNIX for Beginners Questions & Answers

Calculating correlations across columns in awk

Hello, I came across a previous thread "awk-calculating-simple-correlation-rows" which calculated correlations across rows in awk. Code: awk '{ a = 0; for (i = 2; i <= NF; ++i) a += $i; a /= NF-1 b = 0; for (i = 2; i <= NF; ++i) b += ($i - a) ^ 2; b = sqrt(b) if... (7 Replies)
Discussion started by: Ross
7 Replies

LEARN ABOUT OSF1

deckanji

deckanji(5)							File Formats Manual						       deckanji(5)

NAME

       deckanji - A character encoding system (codeset) for Japanese

DESCRIPTION

       The DEC Kanji codeset consists of the following characters: ASCII or JIS X0201 Roman letters Katakana characters specified by JIS X0201 JIS
       X0208 characters User-defined characters (UDC)

       DEC Kanji uses a combination of single-byte data and two-byte data to represent these characters.

       JIS X0201 is a single-byte character set and consists of Roman letters and Katakana characters, which are Japanese  phonetic  symbols.  The
       Roman  letters  defined in the JIS X0201-1976 standard are the same as ASCII letters. (For information on the ASCII character set, refer to
       ascii(5).) However, JIS X0201 replaces the ASCII tilde (~) character with a horizontal bar (located at the upper part of a character cell),
       and replaces the backslash () with the Japanese currency sign (Yen).

       The JIS XO208 standard specifies two-byte character values that represent a variety of characters, including ideographic symbols.

   DEC Kanji Encoding
       All ASCII characters are represented by single-byte 7-bit values in DEC Kanji. That is, the most significant bit (MSB) is always set off in
       the byte that represents an ASCII character. The Roman letters and the Katakana characters specified by JIS X0201 are also single-byte val-
       ues in which the most significant bit (MSB) is set off and on, respectively.

       The  code  table  for JIS X0208 characters is divided into 94 rows, numbered from 1 to 94. Each row has 94 columns, also numbered from 1 to
       94. JIS X0208 defines a total of 6877 characters which include: Special symbols, in rows 1 and 2 Numerals and Roman letters, in row 3 Hira-
       gana  characters, in row 4 Katakana characters, in row 5 Greek letters, in row 6 Russian letters, in row 7 Symbols for drawing graphs, dia-
       grams, and lines, in row 8 First level Kanji characters, in rows 16 to 47 Second level Kanji characters, in rows 48 to 84

       To comply with the JIS X0208 standard, each JIS X0208 character is a two-byte value in the DEC Kanji codeset. The MSB of both the first and
       second bytes is always set on to distinguish JIS X0208 characters from ASCII/JIS Roman or user-defined characters.

       For each JIS X0208 character, The first byte of the two-byte value determines the row number and the second determines the column number in
       the JIS X0208 code table. The following formula shows the code value for a JIS X0208 character in relation to its row and column numbers:

       1st byte = A0 + Row number
       2nd byte = A0 + Column number

       For example, if a character is positioned at the first column of the 36th row, its code value is C4A1, which is calculated as follows:

       1st byte = A0 (hex) + 36 = C4 (hex)
       2nd byte = A0 (hex) + 01 = A1 (hex)

       For user-defined character (UDC) definitions, DEC Kanji provides an area of 2914 positions (from row 1 to row 31). Each UDC is  represented
       by  a two-byte value, just like a JIS X0208 character value. However, the MSB of the second byte of a UDC is set off to distinguish it from
       a JIS X0208 character. The code range of the UDC area is A121 to BF7E.

       The following formula calculates the code of a UDC in relation to its row and column numbers:

       1st byte = A0 + Row number
       2nd byte = 20 + Column number

       For example, if a UDC is positioned at the first column of the 16th row, its code value is B021, which is calculated as follows:

       1st byte = A0 (hex) + 16 = B0 (hex)
       2nd byte = 20 (hex) + 01 = 21 (hex)

   Codeset Conversion
       The following codeset converter pairs are available for converting Japanese characters between deckanji and other encoding formats.   Refer
       to  iconv_intro(5)  for an introduction to codeset conversion. For more information about the other codeset for which deckanji is the input
       or output, see the reference page specified in the list item.  eucJP_deckanji, deckanji_eucJP

	      Converting from and to Japanese Extended UNIX Code: eucJP(5).  iso-2022-jp_deckanji, deckanji_iso-2022-jp

	      Converting from and to the ISO 2022-JP codeset: iso2022jp(5).  iso-2022-jpext_deckanji, deckanji_iso-2022-jpext

	      Converting from and to the ISO 2022-JPexp codeset: iso2022jp(5).	JIS7_deckanji, deckanji_JIS7

	      Converting from and to the JIS7 codeset: jiskanji(5).  sdeckanji_deckanji, deckanji_sdeckanji

	      Converting from and to the Super DEC Kanji codeset: sdeckanji(5).  SJIS_deckanji, deckanji_SJIS

	      Converting from and to the Shift JIS codeset: shiftjis(5).

	      Note that Shift JIS encoding is equivalent to the Microsoft code-page format used on PCs for Japanese. Therefore, you can use  these
	      converters to convert data between DEC Kanji and PC code-page format. For information on PC code pages, see code_page(5) UCS-2_deck-
	      anji, deckanji_UCS-2

	      Converting from and to UCS-2 format: Unicode(5).	UCS-4_deckanji, deckanji_UCS-4

	      Converting from and to UCS-4 format: Unicode(5).	UTF-8_deckanji, deckanji_UTF-8

	      Converting from and to UTF--8 format: Unicode(5).

   Japanese Fonts
       The operating system provides the following Japanese bitmap fonts in various sizes and typefaces for 75dpi and 100dpi  (dot-per-inch)  dis-
       play devices: JIS X0201-1976 characters (Gothic family):

	      -jdecw-gothic-medium-r-normal--8-80-75-75-m-40-jisx0201.1976-0	  -jdecw-gothic-medium-r-normal--14-140-75-75-m-70-jisx0201.1976-0
	      -jdecw-gothic-medium-r-normal--12-120-75-75-m-60-jisx0201.1976-0	 -jdecw-gothic-medium-r-normal--24-240-75-75-m-120-jisx0201.1976-0
	      -jdecw-gothic-medium-r-normal--10-100-75-75-m-50-jisx0201.1976-0	  -jdecw-gothic-medium-r-normal--18-180-75-75-m-90-jisx0201.1976-0
	      -jdecw-gothic-medium-r-normal--17-120-100-100-m-85-jisx0201.1976-0				       -jdecw-gothic-medium-r-nor-
	      mal--34-240-100-100-m-170-jisx0201.1976-0 -jdecw-gothic-medium-r-normal--14-100-100-100-m-70-jisx0201.1976-0 -jdecw-gothic-medium-r-
	      normal--25-180-100-100-m-125-jisx0201.1976-0   -jdecw-gothic-medium-r-normal--20-140-100-100-m-100-jisx0201.1976-0    -jdecw-gothic-
	      medium-r-normal--11-80-100-100-m-55-jisx0201.1976-0 JIS X0201-1976 characters (Kmenu family)

	      -jdecw-kmenu-medium-r-normal--12-120-75-75-p-70-jisx0201.1976-0	 -jdecw-kmenu-medium-r-normal--17-120-100-100-p-85-jisx0201.1976-0
	      JIS X0201-1976 characters (Mincho family)

	      -jdecw-mincho-medium-r-normal--8-80-75-75-m-40-jisx0201.1976-0	  -jdecw-mincho-medium-r-normal--14-140-75-75-m-70-jisx0201.1976-0
	      -jdecw-mincho-medium-r-normal--24-240-75-75-m-120-jisx0201.1976-0   -jdecw-mincho-medium-r-normal--10-100-75-75-m-50-jisx0201.1976-0
	      -jdecw-mincho-medium-r-normal--18-180-75-75-m-90-jisx0201.1976-0	-jdecw-mincho-medium-r-normal--17-120-100-100-m-85-jisx0201.1976-0
	      -jdecw-mincho-medium-r-normal--34-240-100-100-m-170-jisx0201.1976-0				       -jdecw-mincho-medium-r-nor-
	      mal--14-100-100-100-m-70-jisx0201.1976-0 -jdecw-mincho-medium-r-normal--25-180-100-100-m-125-jisx0201.1976-0 -jdecw-mincho-medium-r-
	      normal--20-140-100-100-m-100-jisx0201.1976-0  -jdecw-mincho-medium-r-normal--11-80-100-100-m-55-jisx0201.1976-0 JIS X0201-1976 char-
	      acters (Screen family)

	      -jdecw-screen-medium-r-normal--24-240-75-75-m-120-jisx0201-romankana				       -jdecw-screen-medium-r-nor-
	      mal--18-180-75-75-m-80-jisx0201-romankana  -jdecw-screen-medium-r-normal--14-140-75-75-m-70-jisx0201-romankana -jdecw-screen-medium-
	      r-normal--10-100-75-75-m-50-jisx0201-romankana JIS X0208 characters (Gothic family)

	      -jdecw-gothic-medium-r-normal--14-140-75-75-m-140-jisx0208.1983-1  -jdecw-gothic-medium-r-normal--12-120-75-75-m-120-jisx0208.1983-1
	      -jdecw-gothic-medium-r-normal--24-240-75-75-m-240-jisx0208.1983-1  -jdecw-gothic-medium-r-normal--10-100-75-75-m-100-jisx0208.1983-1
	      -jdecw-gothic-medium-r-normal--18-180-75-75-m-180-jisx0208.1983-1     -jdecw-gothic-medium-r-normal--8-80-75-75-m-80-jisx0208.1983-1
	      -jdecw-gothic-medium-r-normal--17-120-100-100-m-170-jisx0208.1983-1				       -jdecw-gothic-medium-r-nor-
	      mal--34-240-100-100-m-340-jisx0208.1983-1 -jdecw-gothic-medium-r-normal--14-100-100-100-m-140-jisx0208.1983-1  -jdecw-gothic-medium-
	      r-normal--25-180-100-100-m-250-jisx0208.1983-1   -jdecw-gothic-medium-r-normal--20-140-100-100-m-200-jisx0208.1983-1  -jdecw-gothic-
	      medium-r-normal--11-80-100-100-m-110-jisx0208.1983-1 JIS X0208 characters (Mincho family)

	      -jdecw-mincho-medium-r-normal--14-140-75-75-m-140-jisx0208.1983-1   jdecw-mincho-medium-r-normal--12-120-75-75-m-120-jisx0208.1983-1
	      -jdecw-mincho-medium-r-normal--24-240-75-75-m-240-jisx0208.1983-1  -jdecw-mincho-medium-r-normal--10-100-75-75-m-100-jisx0208.1983-1
	      -jdecw-mincho-medium-r-normal--18-180-75-75-m-180-jisx0208.1983-1     -jdecw-mincho-medium-r-normal--8-80-75-75-m-80-jisx0208.1983-1
	      -jdecw-mincho-medium-r-normal--17-120-100-100-m-170-jisx0208.1983-1				       -jdecw-mincho-medium-r-nor-
	      mal--34-240-100-100-m-340-jisx0208.1983-1 -jdecw-mincho-medium-r-normal--14-100-100-100-m-140-jisx0208.1983-1  -jdecw-mincho-medium-
	      r-normal--25-180-100-100-m-250-jisx0208.1983-1   -jdecw-mincho-medium-r-normal--20-140-100-100-m-200-jisx0208.1983-1  -jdecw-mincho-
	      medium-r-normal--11-80-100-100-m-110-jisx0208.1983-1 JIS X0208 characters (Screen family)

	      -jdecw-screen-medium-r-normal--24-240-75-75-m-240-jisx0208-kanji00				       -jdecw-screen-medium-r-nor-
	      mal--10-100-75-75-m-100-jisx0208-kanji00	-jdecw-screen-medium-r-normal--18-180-75-75-m-160-jisx0208-kanji00 -jdecw-screen-medium-r-
	      normal--16-160-75-75-m-160-jisx0208-kanji00 -jdecw-screen-medium-r-normal--14-140-75-75-m-140-jisx0208-kanji00 -jdecw-screen-medium-
	      r-normal--24-240-75-75-m-240-jisx0208-kanji11    -jdecw-screen-medium-r-normal--10-100-75-75-m-100-jisx0208-kanji11   -jdecw-screen-
	      medium-r-normal--18-180-75-75-m-160-jisx0208-kanji11 -jdecw-screen-medium-r-normal--14-140-75-75-m-140-jisx0208-kanji11

       For printers, the operating system provides only Japanese fonts that are printer-resident; that is, there are no Japanese fonts that can be
       dynamically down-loaded to the printer. Refer to i18n_printing(5) for general information on printing non-English text.

SEE ALSO

       Commands: locale(1)

       Others:	ascii(5),  code_page(5), eucJP(5), i18n_intro(5), i18n_printing(5), iconv_intro(5), iso2022jp(5), Japanese(5), jiskanji(5), sdeck-
       anji(5), shiftjis(5), Unicode(5)

																       deckanji(5)
All times are GMT -4. The time now is 11:07 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy