Visit Our UNIX and Linux User Community


awk multiple files


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers awk multiple files
# 1  
Old 07-30-2009
awk multiple files

Hi there!
I'm with a problem because I'm working on a script to do a calculation of a mean using multiple files with 3 columns and N-lines.
My input are like this

File1
Code:
0001 0023  5
0120 0376  6
0412 3412  3
...

File2
Code:
0001 0023  3
0120 0376  nan
0412 3412  7
...

File3
Code:
...

Notice that all the files have the same 2 first columns and only the 3d is variable. I need to read all the files in a row and make the mean of the 3d column for every line.
The nan is "not a number" so it can't go on the calculus.

Output
Code:
0001 0023  4
 0120 0376  6
 0412 3412  5

can Anyone give me a help?

I'm trying somthing like this:

Code:
for i in $(seq 1 3); do awk -f t5.awk -v facc=acc$((i-1)).txt t$i.dat > acc$i.txt; rm acc$((i-1)).txt; done

(in the shell)

and the script
Code:
awk '{
#criar o vector
BEGIN {
#variavel de contagem
i = 1
#ciclo que vai buscar a variavel facc
#cada linha  indexada ao vector a
while ((getline<facc)>0) {
    a[i]=$0
    i++
}
i = 1
soma +=$3
}
{ print a[i] " " soma
  i++
}'

but this is for bilding a matrix of all the columns in $3 and that doest help because it creates a file of ~~5G and I dont have that kind of resourses!


Use CODE-tags next time you post code, data or logs to enhance readability and to preserve formatting like indention etc., ty.

Last edited by zaxxon; 07-30-2009 at 11:41 AM..
# 2  
Old 07-30-2009
Do you mean something like this?

Code:
awk 'END {
  for (k in sum) 
    print k, sum[k] / count[k]
  }
{ 
  sum[$1FS$2] += $3 == "nan" ? 0 : $3
  count[$1FS$2]++ 
  }' file[1-3]

Some awk implementations will cast to 0 the string nan implicitly:

Code:
awk 'END {
  for (k in sum) 
    print k, sum[k] / count[k]
  }
{ 
  sum[$1FS$2] += $3
  count[$1FS$2]++ 
  }' file[1-3]

 

Previous Thread | Next Thread
Test Your Knowledge in Computers #104
Difficulty: Easy
Unix is a family of multitasking, portable, multi-user computer operating systems, which do not have time-sharing capability.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

2. Shell Programming and Scripting

Awk with Multiple files

Hello, I wanted to know the best way to do some matching between two files. I am currently using awk. Example Question: If column 4 from file A is contained in column 1 from file B. Print the whole row from file A of mix of information from both files. i.e. File A ... (1 Reply)
Discussion started by: rafir
1 Replies

3. Shell Programming and Scripting

perform 3 awk commands to multiple files in multiple directories

Hi, I have a directory /home/datasets/ which contains a bunch (720) of subdirectories called hour_1/ hour_2/ etc..etc.. in each of these there is a single text file called (hour_1.txt in hour_1/ , hour_2.txt for hour_2/ etc..etc..) and i would like to do some text processing in them. Each of... (20 Replies)
Discussion started by: amarn
20 Replies

4. UNIX for Dummies Questions & Answers

awk in multiple files

Hello all, I have two files: one that looks like this string1 string2 string3 string6 and another that looks like this Wstring1 apple Wstring2 vegetable Wstring3 mouse Wstring4 (3 Replies)
Discussion started by: FelipeAd
3 Replies

5. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

6. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Hi guys, say I have a few files in a directory (58 text files or somthing) each one contains mulitple strings that I wish to replace with other strings so in these 58 files I'm looking for say the following strings: JAM (replace with BUTTER) BREAD (replace with CRACKER) SCOOP (replace... (19 Replies)
Discussion started by: rich@ardz
19 Replies

7. Shell Programming and Scripting

extract multiple cloumns from multiple files; skip rows and include filenames; awk

Hello, I am trying to write a bash shell script that does the following: 1.Finds all *.txt files within my directory of interest 2. reads each of the files (25 files) one by one (tab-delimited format and have the same data format) 3. skips the first 10 rows of the file 4. extracts and... (4 Replies)
Discussion started by: manishabh
4 Replies

8. Shell Programming and Scripting

Multiple search string in multiple files using awk

Hi, filenames: contains name of list of files to search in. placelist contains the names of places to be searched in all files in "filenames" for i in $(<filenames) do egrep -f placelist $i if ] then echo $i fi done >> outputfile Output i am getting: (0 Replies)
Discussion started by: pinnacle
0 Replies

9. Shell Programming and Scripting

awk with multiple files

I have 2 files first is category: 1:Phone 2:Keyboard 3:Printer 4:Scanner 5:Mouse the second is product: 1:iphone:1:..... 2:blackberry:1:..... 3:Mitsumi:2:..... 4:abc:5:..... 5:def:4:..... 6:noi:3:..... which bold numbers are "foreign key" in product file and they are "primary... (5 Replies)
Discussion started by: WuZun
5 Replies

10. Shell Programming and Scripting

Splitting input files into multiple files through AWK command

Hi, I needs to split *.txt files from single directory depends on the some mutltiple input values. i have wrote the code like below for file in *.txt do grep -i -h "value1|value2" $file > $file; done. My requirment is more input values needs to be given in grep; let us say 50... (3 Replies)
Discussion started by: arund_01
3 Replies

Featured Tech Videos