Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Use awk to read multiple files twice
Top Forums UNIX for Advanced & Expert Users Use awk to read multiple files twice Post 302606589 by ctsgnb on Monday 12th of March 2012 06:44:49 AM
Old 03-12-2012
Code:
$ cat myawk
BEGIN{
    FS = ",";
    OFS = "\t";
    glbcnt[""]=0;
    glbacc[""]=0;
    glbprcn[""]=0;
}
{
k[$1]=$2
e[$1":"NR]=$3
n=NR
}
END{
    print "indx","range","deviation","mean","num of elements";
    for(i in k){
        for (j=0;++j<=n;){
            if (e[i":"j]=="") continue
            glbacc[i]+=e[i":"j]
            glbcnt[i]++
        }
    }
    for(o in k){
        for (p=0;++p<=n;){
            if (e[o":"p]=="") continue
            delta[o":"p]=(e[o":"p]-glbacc[o])
            sumdelta[o]+=(delta[o":"p]^2)
        }
    }
    for(d in glbacc){
        if(d=="") continue
        glbacc[d] = glbacc[d]/glbcnt[d];
        drift[d]=sqrt(sumdelta[d]/glbcnt[d]);
        print d,k[d],drift[d],glbacc[d],glbcnt[d];
    }

}

Code:
$ awk -f myawk t1 t2
indx    range    deviation    mean    num of elements
0    0.0-0.1    0.00421142    0.001052    5
1    0.1-0.2    0.0037352    0.000743333    6
2    0.2-0.4    0.012866    0.00214429    7
3    0.9-1.0    0.0295094    0.0098325    4
$

This code assume that the pseudo code of the formula used to calculate the deviation is :

squareroot_of ( sum _of ( square_of(element - mean of elements) ) / number of elements )

Just feel free to adapt to your needs.
Last edited by ctsgnb; 03-12-2012 at 08:10 AM..
This User Gave Thanks to ctsgnb For This Post:
scandy
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to read from multiple files

Hi All, I have list of multiple files with 7 fields all together. Those are being split to exact lines of 20000 each. xaa xab : : : xhx Please advise me how to read from those files and in fact I need to invoke and sql update statement for each inputs values.. Regards, (5 Replies)
Discussion started by: cedrichiu
5 Replies

2. Shell Programming and Scripting

Awk - to test multiple files "read" permission ?

Hi Masters, Iam new to this Forum and this is my first post. My question is: I've some datafiles belongs the type (A, B, C) in the location 'export/home/lokiman ' dataA1.txt dataB28.txt dataC35.txt 1) I've to check the read permission for each file, if it not there then I've to... (1 Reply)
Discussion started by: lokiman
1 Replies

3. Shell Programming and Scripting

How to Read Multiple files in a Shell Script

Hi, Can any one tell me if i can read two files in a shell script... My actual requirement is to read the 1st text file and parse it to get the file code and use this file code to retrieve data from database and print the fetched data in the 2nd text file (I have parsed it and printed the... (2 Replies)
Discussion started by: funonnet
2 Replies

4. Shell Programming and Scripting

Read and edit multiple files using a while loop

Hi all, I would like to simply read a file which lists a number of pathnames and files, then search and replace key strings using a few vi commands: :1,$s/search_str/replace_str/g<return> but I am not sure how to automate the <return> of these vis commands when I am putting this in a... (8 Replies)
Discussion started by: cyberfrog
8 Replies

5. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

6. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

7. Shell Programming and Scripting

read the lines of multiple files

I am trying to create a script which will read 2 files and use the lines of file 1 for each line on file 2. here's my sample code cat $SBox | while read line do cat $Date | while read line do $SCRIPTEXE <line from first file> $2 <line from 2nd file> ... (12 Replies)
Discussion started by: khestoi
12 Replies

8. UNIX for Dummies Questions & Answers

awk - how to read multiple files

Hi, is there a ways to read multiple files in a single awk command? For example: awk -f awk_script file1 file2 file3 I've google it, most of them suggest using FNR. But I don't understand how it works. It will be a great help if someone able to explain it in simple term with some example. (4 Replies)
Discussion started by: KCApple
4 Replies

9. Shell Programming and Scripting

Script to read multiple files...

I have 7 text files of varying sizes for each month of System Maintenance done during the 2013 calendar year (Jan. 134 jobs, Feb. 84 jobs, Apr. 594 jobs, May 158 jobs, July 69 jobs, Aug. 1 job, Oct. 102 jobs) and I have another text file which contains everything from those 7 files. Each of the... (8 Replies)
Discussion started by: CyberOptiq
8 Replies

10. UNIX for Beginners Questions & Answers

awk GSUB read field values from multiple text files

My program run without error. The problem I am having. The program isn't outputting field values with the column headers to file.txt. Each of the column headers in file.txt has no data. MEMSIZE SECOND SASFoundation Filename The output results in file.txt should show: ... (1 Reply)
Discussion started by: dellanicholson
1 Replies

LEARN ABOUT NETBSD

prop_array_get_cstring

PROP_ARRAY_UTIL(3)					   BSD Library Functions Manual 					PROP_ARRAY_UTIL(3)

NAME

     prop_array_util, prop_array_get_bool, prop_array_set_bool, prop_array_get_int8, prop_array_get_uint8, prop_array_set_int8,
     prop_array_set_uint8, prop_array_get_int16, prop_array_get_uint16, prop_array_set_int16, prop_array_set_uint16, prop_array_get_int32,
     prop_array_get_uint32, prop_array_set_int32, prop_array_set_uint32, prop_array_get_int64, prop_array_get_uint64, prop_array_set_int64,
     prop_array_set_uint64, prop_array_add_int8, prop_array_add_uint8, prop_array_add_int16, prop_array_add_uint16, prop_array_add_int32,
     prop_array_add_uint32, prop_array_add_int64, prop_array_add_uint64, prop_array_get_cstring, prop_array_set_cstring,
     prop_array_get_cstring_nocopy, prop_array_set_cstring_nocopy, prop_array_add_and_rel -- array property collection object utility functions

LIBRARY

     Property Container Object Library (libprop, -lprop)

SYNOPSIS

     #include <prop/proplib.h>

     bool
     prop_array_get_bool(prop_array_t dict, unsigned int indx, bool *valp);

     bool
     prop_array_set_bool(prop_array_t dict, unsigned int indx, bool val);

     bool
     prop_array_get_int8(prop_array_t dict, unsigned int indx, int8_t *valp);

     bool
     prop_array_get_uint8(prop_array_t dict, unsigned int indx, uint8_t *valp);

     bool
     prop_array_set_int8(prop_array_t dict, unsigned int indx, int8_t val);

     bool
     prop_array_set_uint8(prop_array_t dict, unsigned int indx, uint8_t val);

     bool
     prop_array_get_int16(prop_array_t dict, unsigned int indx, int16_t *valp);

     bool
     prop_array_get_uint16(prop_array_t dict, unsigned int indx, uint16_t *valp);

     bool
     prop_array_set_int16(prop_array_t dict, unsigned int indx, int16_t val);

     bool
     prop_array_set_uint16(prop_array_t dict, unsigned int indx, uint16_t val);

     bool
     prop_array_get_int32(prop_array_t dict, unsigned int indx, int32_t *valp);

     bool
     prop_array_get_uint32(prop_array_t dict, unsigned int indx, uint32_t *valp);

     bool
     prop_array_set_int32(prop_array_t dict, unsigned int indx, int32_t val);

     bool
     prop_array_set_uint32(prop_array_t dict, unsigned int indx, uint32_t val);

     bool
     prop_array_get_int64(prop_array_t dict, unsigned int indx, int64_t *valp);

     bool
     prop_array_get_uint64(prop_array_t dict, unsigned int indx, uint64_t *valp);

     bool
     prop_array_set_int64(prop_array_t dict, unsigned int indx, int64_t val);

     bool
     prop_array_set_uint64(prop_array_t dict, unsigned int indx, uint64_t val);

     bool
     prop_array_set_int32(prop_array_t dict, unsigned int indx, int32_t val);

     bool
     prop_array_set_uint32(prop_array_t dict, unsigned int indx, uint32_t val);

     bool
     prop_array_add_int8(prop_array_t dict, int8_t val);

     bool
     prop_array_add_uint8(prop_array_t dict, uint8_t val);

     bool
     prop_array_add_int16(prop_array_t dict, int16_t val);

     bool
     prop_array_add_uint16(prop_array_t dict, uint16_t val);

     bool
     prop_array_add_int32(prop_array_t dict, int32_t val);

     bool
     prop_array_add_uint32(prop_array_t dict, uint32_t val);

     bool
     prop_array_add_int64(prop_array_t dict, int64_t val);

     bool
     prop_array_add_uint64(prop_array_t dict, uint64_t val);

     bool
     prop_array_get_cstring(prop_array_t dict, unsigned int indx, char **strp);

     bool
     prop_array_set_cstring(prop_array_t dict, unsigned int indx, const char *str);

     bool
     prop_array_get_cstring_nocopy(prop_array_t dict, unsigned int indx, const char **strp);

     bool
     prop_array_set_cstring_nocopy(prop_array_t dict, unsigned int indx, const char *strp);

     bool
     prop_array_add_and_rel(prop_array_t dict, prop_object_t obj);

DESCRIPTION

     The prop_array_util family of functions are provided to make getting and setting values in arrays more convenient in some applications.

     The getters check the type of the returned object and, in some cases, also ensure that the returned value is within the range implied by the
     getter's value type.

     The setters handle object creation and release for the caller.

     The prop_array_get_cstring() function returns dynamically allocated memory.  See prop_string(3) for more information.

     The prop_array_get_cstring_nocopy() and prop_array_set_cstring_nocopy() functions do not copy the string that is set or returned.	See
     prop_string(3) for more information.

     The prop_array_add_and_rel() function adds the object to the end of the array and releases it.  The object is also released on failure.

RETURN VALUES

     The prop_array_util getter functions return true if the object exists in the array and the value is in-range, or false otherwise.

     The prop_array_util setter functions return true if creating the object and storing it in the array is successful, or false otherwise.

SEE ALSO

     prop_array(3), prop_bool(3), prop_number(3), proplib(3)

HISTORY

     The proplib property container object library first appeared in NetBSD 4.0.

BSD
								  March 12, 2011							       BSD
All times are GMT -4. The time now is 02:44 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy