Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Calculation in Multiple files using awk

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Calculation in Multiple files using awk
# 1  
Old 08-06-2010
Debian Calculation in Multiple files using awk
Hi All,

I have some 10 files named samp1.csv, samp2.csv,... samp10.csv

Each file having the same number of fields like,

Count, field1, field2, field3.

And a source.csv file which has three fields field1, field2, field3.

Now, i want to find the total count by taking the field1, field2, field3 as uniqueness.

Eg:

source.csv
a, b, c
b, c, a
d, e, f

samp1.csv
10, a, b, c
5, b, c, a

samp2.csv
12, a, b, c
4, b, c, a

samp3.csv
2, a, b, c
7, d, e, f
Output should be,

24, a, b, c
9, b, c, a
7, d, e, f

I am trying to achieve this. Will this be done using awk.

Let me know your inputs. Thanks.
Last edited by johnwilliams.sp; 08-06-2010 at 06:08 AM.. Reason: Alignment is not proper
# 2  
Old 08-06-2010
Hi,

try

Code:
awk -F, '{gsub(/ */,"",$0);split($0,t,",");a[t[2]", "t[3]", "t[4]]+=$1}END{for (i in a) print a[i],i}' file1 file2 file3

Output:
Code:
24 a, b, c
7 d, e, f
9 b, c, a

HTH Chris
# 3  
Old 08-06-2010
Hi Chris,

Can you please explain the code. Smilie
# 4  
Old 08-06-2010
O.k.

Code:
awk -F,  # make "," the new field delimiter
'{gsub(/ */,"",$0); # the format of your data is not consistent
                    # -> delete all blanks
split($0,t,",");    # split the current line at each comma
                    # store the values in an array called t
a[t[2]", "t[3]", "t[4]]+=$1} # now the values in t don't contain any spaces of commas any more
                    # so i construct a new string t[2]", "t[3]", "t[4] of the elements in t
                    # and use it as the name for the hash entry
                    # if this string is encountered again, add 1 to the value associated with the hash name
END{for (i in a) print a[i],i}' # when all files are processed
                    # print out first the number stored in the hash
                    # then the corresponding hash string
file1 file2 file3   # names of the three processed files

This User Gave Thanks to Christoph Spohr For This Post:
johnwilliams.sp
# 5  
Old 08-09-2010
Hi Chris,

I have one query here. In a csv file all fields are delimited by comma(,).

But in the code you have given,

Code:
awk -F, ' '

why does it not take each individaual field in variable $0,$1,$2.

Instead you are splitting using a split function.

Let me know if my query is vague. Smilie
# 6  
Old 08-09-2010
Simply because it didn't work that way, so i tried something else. You can't use $2..$4 to create the hash table value as your input file contains different amounts of spaces. You would get different entries for different amounts of spaces. So at first i have to delete unnecessary spaces, but then i cannot use the old field variables anymore. That's why i have to split the new version of $0 again. But i am not awk-expert.
# 7  
Old 08-09-2010
Its works for me anyway. Just curious of knowing it.

Thanks for your help. Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk split and awk calculation in the same command

I am trying to run the awk below. My question is when I split the input, then run anotherawk to perform a calculation using that splitas the input there are no issues. When I try to combine them the output is not correct, is the split not working or did I do it wrong? Thank you :). input ... (8 Replies)
Discussion started by: cmccabe
8 Replies

2. Shell Programming and Scripting

Multiple Column Calculation

Hallo Team, I need you you help. I need to sum up all the columns in a .csv file. Lets call the file file1.csv and it looks like below: $ cat file1.csv... (2 Replies)
Discussion started by: kekanap
2 Replies

3. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

4. Shell Programming and Scripting

perform 3 awk commands to multiple files in multiple directories

Hi, I have a directory /home/datasets/ which contains a bunch (720) of subdirectories called hour_1/ hour_2/ etc..etc.. in each of these there is a single text file called (hour_1.txt in hour_1/ , hour_2.txt for hour_2/ etc..etc..) and i would like to do some text processing in them. Each of... (20 Replies)
Discussion started by: amarn
20 Replies

5. Shell Programming and Scripting

echo multiple variable with calculation

$total=500 echo "scale=2; $val1*100/$total" | bc echo "scale=2; $val2*100*100/$total" | bc echo "scale=2; $val3*100/$total" | bc I want to make the above code to be accomplish in a single echo line. For instance output:21.3, 44.2, 51.6 How to achieve that, some one please help, i just... (5 Replies)
Discussion started by: alvin0618
5 Replies

6. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

7. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Hi guys, say I have a few files in a directory (58 text files or somthing) each one contains mulitple strings that I wish to replace with other strings so in these 58 files I'm looking for say the following strings: JAM (replace with BUTTER) BREAD (replace with CRACKER) SCOOP (replace... (19 Replies)
Discussion started by: rich@ardz
19 Replies

8. Shell Programming and Scripting

extract multiple cloumns from multiple files; skip rows and include filenames; awk

Hello, I am trying to write a bash shell script that does the following: 1.Finds all *.txt files within my directory of interest 2. reads each of the files (25 files) one by one (tab-delimited format and have the same data format) 3. skips the first 10 rows of the file 4. extracts and... (4 Replies)
Discussion started by: manishabh
4 Replies

9. Shell Programming and Scripting

Multiple search string in multiple files using awk

Hi, filenames: contains name of list of files to search in. placelist contains the names of places to be searched in all files in "filenames" for i in $(<filenames) do egrep -f placelist $i if ] then echo $i fi done >> outputfile Output i am getting: (0 Replies)
Discussion started by: pinnacle
0 Replies

10. Shell Programming and Scripting

Splitting input files into multiple files through AWK command

Hi, I needs to split *.txt files from single directory depends on the some mutltiple input values. i have wrote the code like below for file in *.txt do grep -i -h "value1|value2" $file > $file; done. My requirment is more input values needs to be given in grep; let us say 50... (3 Replies)
Discussion started by: arund_01
3 Replies
Login or Register to Ask a Question