UNIX for Dummies Questions & Answers

Hi would like to ask you guys any advise regarding my problem

I have this kind of data
file.txt

Code:

111111111,20
111111111,50
222222222,70
333333333,40
444444444,10
444444444,20

I need to get this
file1.txt

Code:

111111111,70
222222222,70
333333333,40
444444444,30

using this code I can obtain the output:

Code:

awk -F, '{a[$1]+=$2;}END{for(i in a)print i","a[i];}' file.txt > file1.txt

the starts when it process my actual file have almost 20 million records.

any other suggestion???

Hello,

kindly use the code tags as per forum rules.

If order doesn't matter for you then following may help.

Code:

awk -F, '{a[$1]+=$2;} END{for(i in a) {print i OFS a[i]}}' OFS=, get_fixed_column_sum_check_test_check12112334

Output will be as follows.

Code:

444444444,30
111111111,70
222222222,70
333333333,40

If in order then please use the following.

Code:

awk -F, '{a[$1]+=$2;} END{for(i in a) {print i OFS a[i]}}' OFS=, get_fixed_column_sum_check_test_check12112334 | sort -t1

Output wil lbe as follows.

Code:

111111111,70
222222222,70
333333333,40
444444444,30

NOTE: where get_fixed_column_sum_check_test_check12112334 is the input file name.


Thanks,
R. Singh
"GOD helps those who help themselves."

I'm still getting this error

Code:

awk: 0602-561 There is not enough memory available now.
 The input line number is 1.36372e+07. The file is test.txt.
 The source line number is 1.

Is your file already sorted?

If sorted, you only need to store the 1st field of the previous line and the actual sum:

Code:

awk -F, '$1!=p1 && NR>1 {print p1","sum; sum=0} {p1=$1; sum+=$2} END {if (NR>0) print p1","sum}' file.txt

Thanks MadeInGermany

this works

by the way can you please explain what happened?

Since you didn't say anything about whether or not your input file was sorted, the earlier suggestions had to make the assumption that lines in your 20,000,000 line file were in random order. Therefore, all of the key values and the sums of the corresponding 2nd fields had to be kept in memory until the entire file had been read. Then the totals could be printed for each of the different keys present in the file. The error messages you got say that awk ran out of memory trying to accumulate all of the data.

Now that we know that all of the lines with a given key are adjacent in your input file, the later scripts could print a sum for each key as soon as a new key was found. Very little memory is required to do that and your script runs much faster because it needs less system resources to get the job done.