Sponsored Content
Top Forums UNIX for Advanced & Expert Users Use awk to read multiple files twice Post 302606627 by ctsgnb on Monday 12th of March 2012 09:42:37 AM
Old 03-12-2012
In one way or another, you will have to do 2 pass since the mean is needed to calculate the deviation.

If you can't do the 2 pass because of memory limitation you can either : split the task into shorter one that the memory can handle and/or go for the use a temporary file that you will then scan to calculated you deviation.

---------- Post updated at 02:42 PM ---------- Previous update was at 02:39 PM ----------

The real point is : achieving calculation & processing of such a data volume should be done at a database level, not at a scripting level.

Last edited by ctsgnb; 03-12-2012 at 11:20 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to read from multiple files

Hi All, I have list of multiple files with 7 fields all together. Those are being split to exact lines of 20000 each. xaa xab : : : xhx Please advise me how to read from those files and in fact I need to invoke and sql update statement for each inputs values.. Regards, (5 Replies)
Discussion started by: cedrichiu
5 Replies

2. Shell Programming and Scripting

Awk - to test multiple files "read" permission ?

Hi Masters, Iam new to this Forum and this is my first post. My question is: I've some datafiles belongs the type (A, B, C) in the location 'export/home/lokiman ' dataA1.txt dataB28.txt dataC35.txt 1) I've to check the read permission for each file, if it not there then I've to... (1 Reply)
Discussion started by: lokiman
1 Replies

3. Shell Programming and Scripting

How to Read Multiple files in a Shell Script

Hi, Can any one tell me if i can read two files in a shell script... My actual requirement is to read the 1st text file and parse it to get the file code and use this file code to retrieve data from database and print the fetched data in the 2nd text file (I have parsed it and printed the... (2 Replies)
Discussion started by: funonnet
2 Replies

4. Shell Programming and Scripting

Read and edit multiple files using a while loop

Hi all, I would like to simply read a file which lists a number of pathnames and files, then search and replace key strings using a few vi commands: :1,$s/search_str/replace_str/g<return> but I am not sure how to automate the <return> of these vis commands when I am putting this in a... (8 Replies)
Discussion started by: cyberfrog
8 Replies

5. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

6. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

7. Shell Programming and Scripting

read the lines of multiple files

I am trying to create a script which will read 2 files and use the lines of file 1 for each line on file 2. here's my sample code cat $SBox | while read line do cat $Date | while read line do $SCRIPTEXE <line from first file> $2 <line from 2nd file> ... (12 Replies)
Discussion started by: khestoi
12 Replies

8. UNIX for Dummies Questions & Answers

awk - how to read multiple files

Hi, is there a ways to read multiple files in a single awk command? For example: awk -f awk_script file1 file2 file3 I've google it, most of them suggest using FNR. But I don't understand how it works. It will be a great help if someone able to explain it in simple term with some example. (4 Replies)
Discussion started by: KCApple
4 Replies

9. Shell Programming and Scripting

Script to read multiple files...

I have 7 text files of varying sizes for each month of System Maintenance done during the 2013 calendar year (Jan. 134 jobs, Feb. 84 jobs, Apr. 594 jobs, May 158 jobs, July 69 jobs, Aug. 1 job, Oct. 102 jobs) and I have another text file which contains everything from those 7 files. Each of the... (8 Replies)
Discussion started by: CyberOptiq
8 Replies

10. UNIX for Beginners Questions & Answers

awk GSUB read field values from multiple text files

My program run without error. The problem I am having. The program isn't outputting field values with the column headers to file.txt. Each of the column headers in file.txt has no data. MEMSIZE SECOND SASFoundation Filename The output results in file.txt should show: ... (1 Reply)
Discussion started by: dellanicholson
1 Replies
CFETOOLUPDATE(8)					User Contributed Perl Documentation					  CFETOOLUPDATE(8)

NAME
cfetoolupdate - Update the database with a new value SYNOPSIS
cfetool update name --value|-V value [--path|-p directory name] [--time|-t seconds] [--daily|-d] [--weekly|-w] [--yearly|-y] [--cfenvd|-c] [--histograms|-H] [--verbose|-v] [--help|-h] DESCRIPTION
Takes a new value and uses it to update the database specified by name, located at the path specified by the -p argument, or the current working directory if the -p argument is omitted. The value will be associated with the current time, unless the -t option is given. In order for cfetool to function properly, this command should be issued at least once during each interval specified by the --step option of "cfetool create". The -d, -w and -y options specify the databases to update. If all three options are omitted, only the weekly database will be updated. OPTIONS
--value|-v value Specifies a new value to update the database with. --path|-p directory name The directory in which the database specified by name can be found. --time|-t The time the value was collected, in seconds since epoch (January 1st, 1970). If this argument is omitted, the current time will be used. --daily|-d Update the daily averages database. --weekly|-w Update the weekly averages database. --yearly|-y Update the yearly averages database. --cfenvd|-c Communicate with cfengine. If this option is specified, cfetool will define classes understandable by cfagent. For more information, refer to the documentation for cfenvd. --histograms|-H Collect histogram information about the probability distribiution of fluctuations about the mean value over all times of day. Cfetool won't write anything to this file for the first four weeks, as it has to have reliable mean values in order to accurately calculate it. --verbose|-v Print details of the command's execution to the standard output stream. --help|-h Print a short help message and then exit. OUTPUT
Before exiting, "cfetool update" will print one line to the standard output stream in the following format: yrly=ynum,wkly=wnum,dly=dnum ynum, wnum and dnum will be either the number 0 if the corresponding database was not updated, or a code indicating the state of the given statistic, as compared to an average of equivalent earlier times, as specified below: code high|low|normal meaning ------------------------------------------------------------- -2 - no sigma variation ------------------------------------------------------------- -4 low within noise threshold, and within -5 normal 2 standard deviations from -6 high expected value ------------------------------------------------------------- -14 low microanomaly: within noise -15 normal threshold, but 2 or more standard -16 high deviations from expected value ------------------------------------------------------------- -24 low normal; within 1 standard deviation -25 normal from the expected value -26 high ------------------------------------------------------------- -34 low dev1; more than 1 standard -35 normal deviation from the expected -36 high value ------------------------------------------------------------ -44 low dev2; more than 2 standard -45 normal deviations from the expected -46 high value ------------------------------------------------------------- -54 low anomaly; more than 3 standard -55 normal deviations from the expected -56 high value Where "low" indicates that the current value is below both the expected value for the current time position, and the global average value. "high" indicates that the current value is above those values. "normal" indicates that the current value is within the range of expected values. "cfetool update" also exits with a code corresponding to the above table. If more than one database is being updated, the most negative result from the updates is returned, and the individual results must be obtained from the standard output stream, as described above. EXAMPLE
% cfetool update temperature --path /my/path --value 14.5 --histograms yrly=0,wkly=-6,dly=0 Updates the weekly temperature database and histogram files located in /my/path/ with the value 14.5 using the current time as the update time. The output indicates that only a weekly database was updated, and the new value given was within cfetool's noise threshold, and also within 2 standard deviations of the previous average stored in the database. AUTHORS
The code and documentation were contributed by Stanford Linear Accelerator Center, a department of Stanford University. This documentation was written by Elizabeth Cassell <e_a_c@mailsnare.net> and Alf Wachsmann <alfw@slac.stanford.edu> COPYRIGHT AND DISCLAIMER
Copyright 2004 Alf Wachsmann <alfw@slac.stanford.edu> and Elizabeth Cassell <e_a_c@mailsnare.net> All rights reserved. perl v5.8.4 2004-09-21 CFETOOLUPDATE(8)
All times are GMT -4. The time now is 02:34 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy