awk script processing data from 2 files Post: 302460556

Sponsored Content

Top Forums Shell Programming and Scripting awk script processing data from 2 files Post 302460556 by Alice236 on Thursday 7th of October 2010 04:19:07 AM

10-07-2010

Registered User

awk script processing data from 2 files

Hi!
I have 2 files containing data that I need to process at the same time, I have problems in reading a different number of lines from the different files.
Here is an explanation of what I need to do (possibly with an awk script).

File "samples.txt" contains data in the format:
time_instant measure

HTML Code:

903.0 -
906.43 18.4
912.7 17.5
918.05 11.2

File "time.txt" contains data in the format
time_instant position

HTML Code:

900 out
901 out
902 out
903 in
904 in
905 in
906 in
907 in
908 out
909 out
910 out
911 in
912 in
913 in

I need to compute temporal averages of the measures in samples.txt, but the temporal averages must be related only to the time instants in which the position in time.txt is "in".

So, I need at the end to have:

a/b
where a=sum(measure_i*duration_measure_i)
b=total_measurement_duration

- measure_i is the second field in every line of samples.txt
- duration_measure_i is the difference between the 1st fields in two consecutive lines in samples.txt and eventually the duration of the period in which the position in time.txt was "out"

So, for example I would do for the first measurement (that is on the second line in samples.txt, since the 1st line only tells me when measurements start):
a=18.4*(906.43-903.0-0)
b=(906.43-903.0-0)

Then for the second:
a+=17.5*(912.7-906.43-2)
b+=(912.7-906.43-2)

I am quite new to awk and don't know how to read two lines at a time from samples.txt and search in time.txt the lines that are between the time instants written from samples.txt

Any suggestion? Thank you very much!!

Alice236

View Public Profile for Alice236

Find all posts by Alice236

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

single output of awk script processing multiple files

Helllo UNIX Forum :) Since I am posting on this board, yes, I am new to UNIX! I read a copy of "UNIX made easy" from 1990, which felt like a making a "computer-science time jump" backwards ;) So, basically I have some sort of understanding what the basic concept is. Problem Description:...

2. Shell Programming and Scripting

Help with data processing, maybe awk

I have a file, first 5 columns are very normal, like "1107",106027,71400,"Y","BIOLOGY",, however, the 6th columns, the user can put comments, anything, just any characters, like new line, double quote, single quote, whatever from the keyboard, like"Please load my previous SOM597G course content in...

3. Shell Programming and Scripting

Processing files using awk

Hi I have files in our UNIX directory like the below -rw-r--r-- 1 devinfo devsupp 872 Sep 14 02:09 IMGBTREE27309_12272_11_1_0_FK.idx0 -rw-r--r-- 1 devinfo devsupp 872 Sep 14 02:09 IMGBTREE27309_12272_11_0_0_PK.idx0 -rw-r--r-- 1 devinfo devsupp 432 Sep 14...

4. Shell Programming and Scripting

Perl script required for processing the data

I have following result.log file (always has 2 lines) which I need to process, cat result.log name.cmd.method,"result","abc","xyz"; name="hello,mine.12345,"&"tree"&" xyz "&" tree "&" xyz", data="way,"&" 1"&"rate-me"&"1"&"rate-me",str="",ret=""; now I need to extract the strings/data as...

5. Shell Programming and Scripting

Data processing using awk

Hello, I have some bitrate data in a csv which is in an odd format and is difficult to process in Excel when I have thousands of rows. Therefore, I was thinking of doing this in bash and using awk as the primary application except that due to its complication, I'm a little stuck. ...

6. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M...

7. Shell Programming and Scripting

awk processing of variable number of fields data file

Hy! I need to post-process some data files which have variable (and periodic) number of fields. For example, I need to square (data -> data*data) the folowing data file: -5.34281E-28 -3.69822E-29 8.19128E-29 9.55444E-29 8.16494E-29 6.23125E-29 4.42106E-29 2.94592E-29 1.84841E-29 ...

8. Shell Programming and Scripting

Converting text files to xls through awk script for specific data format

Dear Friends, I am in urgent need for awk/sed/sh script for converting a specific data format (.txt) to .xls. The input is as follows: >gi|1234|ref| Query = 1 - 65, Target = 1677 - 1733 Score = 8.38, E = 0.6529, P = 0.0001513, GC = 46 fd sdfsdfsdfsdf fsdfdsfdfdfdfdfdf...

9. Shell Programming and Scripting

Passing multiple files to awk for processing in bash script

Hi, I'm using awk command in bash script. I'm able to pass multiple files to awk for processing.The code i can use is as below(sample code) #!/bin/bash awk -F "," 'BEGIN { ... ... ... }' file1 file2 file3 In the above code i'm passing the file names manually and it is fine till my...

10. UNIX for Beginners Questions & Answers

Processing files one by one using data from pipe

Hi guys, I receive a list from pipe (with fixed number of lines) like this: name1 name2 name3 And in my ./ folder I have three files: 01-oldname.test 02-someoldname.test 03-evenoldername.test How to rename files one by one using while read? Desired result: 01-name1.test 02-name2.test...

LEARN ABOUT LINUX

igawk

IGAWK(1)							 Utility Commands							  IGAWK(1)

NAME

       igawk - gawk with include files

SYNOPSIS

       igawk [ all gawk options ] -f program-file [ -- ] file ...
       igawk [ all gawk options ] [ -- ] program-text file ...

DESCRIPTION

       Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1).

       AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like

	      @include getopt.awk

       in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path.

OPTIONS

       See gawk(1) for a full description of the AWK language and the options that gawk supports.

EXAMPLES

       cat << EOF > test.awk
       @include getopt.awk

       BEGIN {
	    while (getopt(ARGC, ARGV, "am:q") != -1)
		 ...
       }
       EOF

       igawk -f test.awk

SEE ALSO

       gawk(1)

       Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995.

AUTHOR

       Arnold Robbins (arnold@skeeve.com).

Free Software Foundation					    Nov 3 1999								  IGAWK(1)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

single output of awk script processing multiple files

Discussion started by: Kasimir

2. Shell Programming and Scripting

Help with data processing, maybe awk

Discussion started by: freelong

3. Shell Programming and Scripting

Processing files using awk

Discussion started by: rbmuruga

4. Shell Programming and Scripting

Perl script required for processing the data

Discussion started by: perlDiva