Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Dealing with Empty files, AWK and Loops Post 302652045 by A-V on Wednesday 6th of June 2012 12:53:31 PM
Old 06-06-2012
you might be right I am experimenting with it so might not be correct

I get a set of files that have data per-line. they are either words or numbers separated by space.

what I think I was doing is to get the count of the words, or numbers per line using awk NF
then calculate the mean and variance of these lines per file
and finally combine all of them into one single file for mean and variance

Quote:
step 1
hello this is me
hello this is me again

step 2 using awk nf
4
5

step 3
calculate mean and variance for all files in the directory

step 4
put them all in one file
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

While loops and awk

I am trying to make a script that will replace backslashes in a file, but only if the occurance is a pathname. In the file, there are a lot of regular expressions as well, so I'm trying to preserve the integrity of those regular expressions, but convert Windows relative paths. I'm using bash and... (1 Reply)
Discussion started by: Loriel
1 Replies

2. Shell Programming and Scripting

perl: When dealing with files that do not exist

I have a process run weekly where I must convert data formats for about thirty files. I read a text file that provides all of the filenames and switch settings. My perl code is: for ($j = 1; $j <= $k; $j++) { open(FIN2,$fin2) || die "open: $!"; do other stuff } Every once in... (2 Replies)
Discussion started by: joeyg
2 Replies

3. Shell Programming and Scripting

Dealing with log files

Hi , My requirement is that i need to search for a number of strings in a log file and print them with line numbers.The search should be date wise. The sample log file is : Jan 17 02:45:34 srim6165 MQSIv500: (UKBRKR1P_B.LZ_ BENCHMARKS)BIP2648E: Message backed out to a queue; node... (6 Replies)
Discussion started by: charudpss
6 Replies

4. Shell Programming and Scripting

Dealing with files with spaces in the name

Hello, I'm a computer science major and I'm having problems dealing with file names with spaces in them. Particularly I'm saving a file name in a variable and then using the variable in a compare function i.e. a='te xt.txt' b='file2.txt' cmp $a $b If anyone could help me with this particular... (10 Replies)
Discussion started by: jakethegreycat
10 Replies

5. Shell Programming and Scripting

Iterating over subdirectories and dealing with files within them

Hello, I am working on a coding project for a class and to test the program I have created, I have come up with 100 different test cases. The program takes four text files as input, so each of the test cases is contained in a folder with four files. I have a folder called 'tests', within which... (1 Reply)
Discussion started by: dpryor
1 Replies

6. UNIX Desktop Questions & Answers

awk using 2 input files instead of while loops

Hi Friends, I have two files as input with data that looks like this: file1.txt 1 2 3 4 file2.txt a,aa b,bb c,cc d,dd e,ee f,ff instead of me doing 2 while loops to get the combinations while read line_file1 (2 Replies)
Discussion started by: kokoro
2 Replies

7. UNIX for Dummies Questions & Answers

Dealing with Double Loops, Arrays and GREP

Can someone please help me to learn how to deal with loops, arrays and grep? I have two arrays (lets say I and j) each in a separate file And have file with lines of data I need to extract, such as Ruby Smith: some text here Ruby Smith: some other text here Ruby Brown: some text here Ruby... (10 Replies)
Discussion started by: A-V
10 Replies

8. Shell Programming and Scripting

Dealing with multiple files

Korn Shell I have hundreds of small files like below created every day. A midnight cron job moves them to the location /u04/temp/logs But sometimes I have to manually move these files based a certain dates or time. I have two basic requirements 1.Using mv command I want to move all .dat... (2 Replies)
Discussion started by: kraljic
2 Replies

9. Shell Programming and Scripting

Awk: Dealing with whitespace in associative array indicies

Is there a reliable way to deal with whitespace in array indicies? I am trying to annotate fails in a database using a table of known fails. In a begin block I have code like this: # Read in Known Fail List getline < "'"$failListFile"'"; getline < "'"$failListFile"'"; getline <... (6 Replies)
Discussion started by: Michael Stora
6 Replies

10. Shell Programming and Scripting

Check file from multiple files is empty using awk

I am passing multiple files in awk & since one of the file is empty(say file3) so the same gets skipped & logic goes for toss. Need suggestion/help in checking and putting additional checks for the same awk -F, 'FNR==1 {++filecounter} filecounter==1 {KRL=$2;next} filecounter==2... (8 Replies)
Discussion started by: siramitsharma
8 Replies
Statistics::OnLine(3pm) 				User Contributed Perl Documentation				   Statistics::OnLine(3pm)

NAME
Statistics::OnLine - Pure Perl implementation of the on-line algorithm to produce statistics SYNOPSIS
use Statistics::OnLine; my $s = Statistics::OnLine->new; my @data = (1, 2, 3, 4, 5); $s->add_data( @data ); $s->add_data( 6, 7 ); $s->add_data( 8 ); print "count = ",$s->count," mean = ",$s->mean," variance = ",$s->variance," variance_n = ", $s->variance_n," skewness = ",$s->skewness," kurtosis = ",$s->kurtosis," "; $s->add_data( ); # does nothing! print "count = ",$s->count," mean = ",$s->mean," variance = ",$s->variance," variance_n = ", $s->variance_n," skewness = ",$s->skewness," kurtosis = ",$s->kurtosis," "; $s->add_data( 9, 10 ); print "count = ",$s->count," mean = ",$s->mean," variance = ",$s->variance," variance_n = ", $s->variance_n," skewness = ",$s->skewness," kurtosis = ",$s->kurtosis," "; DESCRIPTION
This module implements a tool to perform statistic operations on large datasets which, typically, could not fit the memory of the machine, e.g. a stream of data from the network. Once instantiated, an object of the class provide an "add_data" method to add data to the dataset. When the computation of some statistics is required, at some point of the stream, the appropriate method can be called. After the execution of the statistics it is possible to continue to add new data. In turn, the object will continue to update the existing data to provide new statistics. METHODS
new() Creates a new "Statistics::OnLine" object and returns it. add_data(@) Adds new data to the object and updates the internal state of the statistics. The method return the object itself in order to use it in chaining: my $v = $s->add_data( 1, 2, 3, 4 )->variance; clean() Cleans the internal state of the object and resets all the internal statistics. Return the object itself in order to use it in chaining: my $v = $s->clean->add_data( 1, 2, 3, 4 )->variance; count() Returns the actual number or elements inserted and processed by the object. mean() Returns the average of the elements inserted into the system: fract{ sum_1^n{x_i} }{ n } variance() Returns the variance of the element inserted into the system: fract{ sum_1^n{avg - x_i} }{ n - 1 } variance_n() Returns the variance of the element inserted into the system: fract{ sum_1^n{avg - x_i} }{ n } skewness() Returns the skewness (third standardized moment) of the element inserted into the system <http://en.wikipedia.org/wiki/Skewness> kurtosis() Returns the kurtosis (fourth standardized moment) of the element inserted into the system <http://en.wikipedia.org/wiki/Kurtosis> ERROR MESSAGES
The conditions in which the system can return errors, using a "die" are: too few elements to compute function Some functions need a minimum number of elements to be computed: "mean", "variance_n" and "skewness" need at least one element, "variance" at least two and "kurtosis" needs at least four. variance is zero: cannot compute kurtosis|skewness Both kurtosis and skewness need that variance to be greater than zero. THEORY
On-line statistics are based on strong mathematical foundations which transform the standard computations into a sequence of operations that incrementally update with new values the actual ones. There are some referencence in the web. This documentation suggest to start your investigation from <http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Higher-order_statistics>. The linked page provides other useful references on the foundations of the method. CAVEAT
The module is intended to be used in all the situations in which:(1) the number of data elements could be too large with respect the memory of the system, or(2) the elements arrive at different time stamps and intermediate results are needed. If the length of the stream is fixed, all the data elements are present in a single place and there is not need for intermediate results, it could be better to use different modules, for instance Statistics::Lite, to make computations. The reason for this choice is that the module uses a stable approximation, well suited for the use on steams (effectively an on-line algorithm). Using this system on fixed datasets could introduce some (little) approximation. HISTORY
0.02 Corrected typos in documentation 0.01 Initial version of the module AUTHOR
Francesco Nidito COPYRIGHT
Copyright 2009 Francesco Nidito. All rights reserved. This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. SEE ALSO
Statistics::Lite, <http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Higher-order_statistics> perl v5.10.1 2009-09-04 Statistics::OnLine(3pm)
All times are GMT -4. The time now is 03:16 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy