Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

merge similar rows

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting merge similar rows
# 1  
Old 11-27-2009
merge similar rows
I have a large file (10M lines) that contains two columns: a frequency and a string, ex:

3 aaaaa
4 bbbbb
2 ccccc
5 aaaaa
1 ddddd
4 ccccc

I need to merge the lines whose string part is the same, while updating the frequency. The output should look like this:

8 aaaaa
4 bbbbb
5 ccccc
1 ddddd

Could you please help me solve this problem?
# 2  
Old 11-27-2009
you can use awk
Code:
awk '{A[$2]+=$1}END{for (i in A){print A[i]" "i}}' filename

# 3  
Old 11-27-2009
And if you want to order by the second filed as in your example:
Code:
perl -ane'
  $_{$F[1]} += $F[0];
   print map "$_{$_} $_\n", sort keys %_ 
     if eof' infile

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

To group the text (rows) by similar columns-names in a file

As part of some report generation, I've written a script to fetch the values from DB. But, unluckily, for certain Time ranges(1-9.99,10-19.99 etc), I don't have data in DB. In such cases, I would like to write zero (0) instead of empty. The desired output will be exported to csv file. ... (1 Reply)
Discussion started by: kumar_karpuram
1 Replies

2. Shell Programming and Scripting

Bash/shell merge similar lines

Hello, I've been working on a bash script to parse through firewall logs (cisco). I'm nearing the end and have a dilemma. My data looks as such (actual data is several gigs worth of logs - without the headers): sourceIP destinationIP destinationProtocol destinationPort 1.1.1.1 2.2.2.2 ... (2 Replies)
Discussion started by: umang2382
2 Replies

3. Shell Programming and Scripting

Transposing rows to columns with multiple similar lines

Hi, I am trying to transpose rows to columns for thousands of records. The problem is there are records that have the same lines that need to be separated. the input file as below:- ID 1A02_HUMAN AC P01892; O19619; P06338; P10313; P30444; P30445; P30446; P30514; AC Q29680; Q29837;... (2 Replies)
Discussion started by: redse171
2 Replies

4. Shell Programming and Scripting

Merge two files with similar column entries

Hi , I have few files which contains user name and data transfer rate in MBs and this data is collected for year and for each month report is saved in 12 different files I have to merge all the files to prepare the final report Files are as below Similarly I have 10 more files ... (5 Replies)
Discussion started by: pratapsingh
5 Replies

5. UNIX for Dummies Questions & Answers

[diff] hide missing rows, show similar

Hi all! Having the following two csv files: file1 AAA;0000;RED CCC;9900;GREEN file2 AAA;0000;BLACK BBB;0099;BLU What's the correct syntax to hide only the missing rows (BBB,CCC) and show the rows that differ only with last field? I expect something like this: diff <options> file1... (2 Replies)
Discussion started by: Evan
2 Replies

6. Shell Programming and Scripting

Help with merge two file based on similar column content

Input file 1: A1BG A1BG A1BG A1CF A1CF BCAS BCAS A2LD1 A2M A2M HAT . . Input file 2: A1BG All A1CF TEMP (5 Replies)
Discussion started by: perl_beginner
5 Replies

7. UNIX for Dummies Questions & Answers

merge lines within a file that start with a similar pattern

Hello! i have a text file.. which contains the data as follows i want to merge the declarations lines pertaining to one datatype in to a single line as follows i've searched the forum for help.. but couldn't find much help.. how can i do this?? (1 Reply)
Discussion started by: a_ba
1 Replies

8. UNIX for Dummies Questions & Answers

Merge two files with two columns being similar

Hi everyone. How can I merge two files, where each file has 2 columns and the first columns in both files are similar? I want all in a file of 4 columns; join command removes the duplicate columns. 1 Dave 2 Mark 3 Paul 1 Apple 2 Orange 3 Grapes to get it like this in the 3rd file:... (9 Replies)
Discussion started by: Atrisa
9 Replies

9. Shell Programming and Scripting

Urgent : Merge similar lines

Hi, I have a file like this. please notice that ./usr/orders1/order_new_2627 appears more than once, thus needs to be merged. I would like to merge the lines where the first column match so the output should be like this: Please help (2 Replies)
Discussion started by: rakeshou
2 Replies

10. UNIX for Dummies Questions & Answers

match similar rows. uniq?

hi i have data which is in two columns (such as below). i need to compare two rows against each other and if one row matches the other row (except for different case), and their values in the second column are different, then it prints out one of the rows (either is fine). here is an... (5 Replies)
Discussion started by: Streetrcr
5 Replies
Login or Register to Ask a Question