Sponsored Content
Full Discussion: sum the columns of files
Top Forums Shell Programming and Scripting sum the columns of files Post 302487934 by frewise on Friday 14th of January 2011 07:04:09 AM
Old 01-14-2011
sum the columns of files

I have several csv files like this:
file1.csv
1 12 1
2 8 9
3 9 2
4 5 9
...
file2.csv
1 0 1
2 2 3
3 4 1
...
file3.csv
1 0 1
2 4 0
...
I want the result like this
1 12 3
2 14 12
3 13 3
4 5 9
I have a script:
Code:
awk '{a[$1]+=$2;b[$1]+=$3}END{for(i in a)print i"\t"a[i]"\t"b[i]}' *.csv|sort -k1n

Due to the huge size of my datasets, I don't want it run many days, I wonder if there is a much more efficiency approach

Last edited by frewise; 01-14-2011 at 08:38 AM..
This User Gave Thanks to frewise For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sum of three columns

Hi All, I have like this M17XX-050-01 0100000000 QQSSS 0.0000e+00 1.712E+06 1.255E+07 0.0000e+00 0.0000e+00 1.722E+06 1.263E+07 0.0000e+00 ... 0.0000e+00 1.204E+06 8.829E+06 0.0000e+00 M17XX-050-01 0100000000 WWSSS 0.0000e+00 7.564E+03 1.165E+01 0.0000e+00... (6 Replies)
Discussion started by: f_o_555
6 Replies

2. Shell Programming and Scripting

Sum of three columns - in 4N columns file

Hi All, happy new year. I have a file with 4xN columns like 0.0000e+00 0.0000e+00 7.199E+07 7.123E+07 6.976E+07 6.482E+07 5.256E+07 2.523E+07 0.0000e+00 0.0000e+00 8.641E+07 8.550E+07 8.373E+07 7.780E+07 6.309E+07 3.028E+07... (8 Replies)
Discussion started by: f_o_555
8 Replies

3. UNIX for Dummies Questions & Answers

Sum of all columns in all files in one output file

If I have say 4 files like this: File1: 1 3 4 7 7 0 5 7 5 9 1 2 7 4 8 File2: 1 4 6 2 5 7 1 2 3 6 0 3 0 3 8 File3: (5 Replies)
Discussion started by: cosmologist
5 Replies

4. Shell Programming and Scripting

Sum up values of columns in 4 files using shell script

I am new to shell script.I have records like below in 4 different files which have about 10000 records each, all records unique and sorted based on column 2. 1 2 3 4 5 6 --------------------------- SR|1010478|000044590|1|0|0| SR|1014759|000105790|1|0|0| SR|1016609|000108901|1|0|0|... (2 Replies)
Discussion started by: reach.sree@gmai
2 Replies

5. Shell Programming and Scripting

Sum columns

Hi All, I'm new to this forum. So please be patience with me! :) I have a file that looks like this (all rows have the same number of columns): 19 20 30 15 17 38 51 60 74 85 96 07 .... 10 20 44 59 39 88 13 77 30 10 11 12 .... . . . I want to sum the value of first field to all the... (2 Replies)
Discussion started by: Aderson Nascime
2 Replies

6. Shell Programming and Scripting

Evaluate 2 columns, add sum IF two columns match on two rows

Hi all, I know this sounds suspiciously like a homework course; but, it is not. My goal is to take a file, and match my "ID" column to the "Date" column, if those conditions are true, add the total number of minutes worked and place it in this file, while not printing the original rows that I... (6 Replies)
Discussion started by: mtucker6784
6 Replies

7. Shell Programming and Scripting

Get the SUM of TWO columns SEPARATELY by doing GROUP BY on other columns

My File looks like: "|" -> Field separator A|B|C|100|1000 D|E|F|1|2 G|H|I|0|7 D|E|F|1|2 A|B|C|10|10000 G|H|I|0|7 A|B|C|1|100 D|E|F|1|2 I need to do a SUM on Col. 5 and Col.6 by grouping on Col 1,2 & 3 My expected output is: A|B|C|111|11100 (2 Replies)
Discussion started by: machomaddy
2 Replies

8. Shell Programming and Scripting

Evaluate 2 columns, add sum IF two columns satisfy the condition

HI All, I'm embedding SQL query in Script which gives following output: Assignee Group Total ABC Group1 17 PQR Group2 5 PQR Group3 6 XYZ Group1 10 XYZ Group3 5 I have saved the above output in a file. How do i sum up the contents of this output so as to get following output: ... (4 Replies)
Discussion started by: Khushbu
4 Replies

9. Shell Programming and Scripting

Sum of Columns

HI Guys, I gave Input file F.Txt ID H1 H2 H3 H4 H5 A 5 6 7 8 9 B 4 65 4 4 7 C 4 4 4 4 4 D 4 4 4 4 4 Output :- ID H1 H2 H3 H4 H5 Total 17 79 19 20 24 Sum of Each Columns (8 Replies)
Discussion started by: pareshkp
8 Replies

10. UNIX for Beginners Questions & Answers

Group by columns and add sum in new columns

Dear Experts, I have input file which is comma separated, has 4 columns like below, BRAND,COUNTRY,MODEL,COUNT NIKE,USA,DUMMY,5 NIKE,USA,ORIGINAL,10 PUMA,FRANCE,DUMMY,20 PUMA,FRANCE,ORIGINAL,15 ADIDAS,ITALY,DUMMY,50 ADIDAS,ITALY,ORIGINAL,50 SPIKE,CHINA,DUMMY,1O And expected output add... (2 Replies)
Discussion started by: ricky1991
2 Replies
csv2po(1)						      Translate Toolkit 1.9.0							 csv2po(1)

NAME
csv2po - convert Comma-Separated Value (.csv) files to Gettext PO localization files SYNOPSIS
csv2po [--version] [-h|--help] [--manpage] [--progress PROGRESS] [--errorlevel ERRORLEVEL] [-i|--input] INPUT [-x|--exclude EXCLUDE] [-o|--output] OUTPUT [-t|--template TEMPLATE] [--charset CHARSET] [--columnorder] [--duplicates DUPLICATESTYLE] DESCRIPTION
See: http://translate.sourceforge.net/wiki/toolkit/csv2po for examples and usage instructions OPTIONS
--version show program's version number and exit -h/--help show this help message and exit --manpage output a manpage based on the help --progress show progress as: dots, none, bar, names, verbose --errorlevel show errorlevel as: none, message, exception, traceback -i/--input read from INPUT in csv format -x/--exclude exclude names matching EXCLUDE from input paths -o/--output write to OUTPUT in po, pot formats -t/--template read from TEMPLATE in pot, po, pot formats --charset set charset to decode from csv files --columnorder specify the order and position of columns (location,source,target) --duplicates what to do with duplicate strings (identical source text): merge, msgctxt (default: 'msgctxt') Translate Toolkit 1.9.0 csv2po(1)
All times are GMT -4. The time now is 09:11 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy