Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Awk: How to merge duplicate lines and print in a single

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Awk: How to merge duplicate lines and print in a single
# 1  
Old 03-14-2011
Awk: How to merge duplicate lines and print in a single
The input file:
Code:
>cat module1
200611051053                     95
200523457498                     35
200617890187                     57
200726098123                     66
200645676712                     71
200744556590                     68

>cat module2
200645676712                     56
200617890187                     50
200523457498                     29
200726098123                     62
200744556590                     69
200611051053                     90

expecting output
Code:
Reg No                      module1       module2
200611051053                     95        90
200523457498                     35        29
200617890187                     57        50
200726098123                     66        62
200645676712                     71        56
200744556590                     68        69

Many Thanks
Last edited by Scott; 03-14-2011 at 03:00 PM.. Reason: Less formatting, more code tags, please...
# 2  
Old 03-14-2011
Use this to collect the required info in a hash
Code:
{ reg[$1] = reg[$1] $2 }

then in the END print all the stuff using a loop.
Last edited by Scott; 03-14-2011 at 02:55 PM..
This User Gave Thanks to 116@434 For This Post:
winter9
# 3  
Old 03-14-2011
Code:
awk 'FNR==NR{a[$1]=$0;} NR>FNR{print a[$1]? a[$1]" "$2 : $0}' t1.txt t2.txt
200645676712 71 56
200617890187 57 50
200523457498 35 29
200726098123 66 62
200744556590 68 69
200611051053 95 90

This User Gave Thanks to sk1418 For This Post:
winter9
# 4  
Old 03-14-2011
Thanks for the replied,

Can you explain a little bit more, I'm new for awk.

Many Thanks!

---------- Post updated at 12:45 PM ---------- Previous update was at 12:41 PM ----------

How can you read module1 and module2 into a 2 seperate parts with command?
# 5  
Old 03-14-2011
I was referring something similar to this:
Code:
cat file1 file2 | awk ' { arr[$1] = arr[$1] $2 }   END {for (i in arr) print i arr[i] }'

Last edited by Scott; 03-14-2011 at 02:55 PM..
This User Gave Thanks to 116@434 For This Post:
winter9
# 6  
Old 03-14-2011
for the first file, create an array. index is the column1. value is the whole line. after file1 was completely processed. start working on file2, read column1 from file2 as key, if the key is already in array, means found the same key in file1, then append the value with column2. then print.

i think you may want to figure out the relation between NR and FNR when awk processing two files. try 2 give a shot by google.
This User Gave Thanks to sk1418 For This Post:
winter9
# 7  
Old 03-14-2011
or even this way:
Code:
awk ' { arr[$1] = arr[$1] $2 } END {for (i in arr) print i arr[i] }' module1 module2

Last edited by Scott; 03-14-2011 at 02:55 PM.. Reason: Please use code tags.
This User Gave Thanks to 116@434 For This Post:
winter9
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merge multi-lines into one single line using shell script or Linux command

Hi, Can anyone help me for merge the following multi-line log which beginning with a " and line ending with ": into one line. *****Original Log***** 087;2008-12-06;084403;"mc;;SYHLR6AP1D\LNZW;AD-703;1;12475;SYHLR6AP1B;1.1.1.1;0000000062;HGPDI:MSISDN=12345678,APNID=1,EQOSID=365;... (3 Replies)
Discussion started by: rajeshlinux2010
3 Replies

2. Shell Programming and Scripting

Merge multiple lines into a single line

Hi all, I'm relatively new to scripting, I can do pretty basic things. I have a daily log file that looks like: timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; ... (29 Replies)
Discussion started by: dwdnet
29 Replies

3. UNIX for Dummies Questions & Answers

[Solved] How to extract single and duplicate lines from file?

Hi, I need help! I have two files, one containing a list of codes and the other a list of codes and their meaning. I need to extract from file 2 all the codes from file 1 into a new file. These are my files: File1: Metbo Metbo Memar Mth Metbo File2: Metbo Methanoculleus... (3 Replies)
Discussion started by: Lokaps
3 Replies

4. UNIX for Dummies Questions & Answers

Need help combining txt files w/ multiple lines into csv single cell - also need data merge

:confused:Hello -- i just joined the forums. I am a complete noob -- only about 1 week into learning how to program anything... and starting with linux. I am working in Linux terminal. I have a folder with a bunch of txt files. Each file has several lines of html code. I want to combine... (2 Replies)
Discussion started by: jetsetter
2 Replies

5. Shell Programming and Scripting

Print duplicate lines

I have a file where some of the lines are duplicates. How do I use bash to print all the lines that have duplicates? (2 Replies)
Discussion started by: locoroco
2 Replies

6. UNIX for Advanced & Expert Users

Merge a group of lines into single line

Hi Everybody, Below are the contents of the a text file .., SN = 8 MSI = 405027002277133 IKVALUE = DE6AA6A11D42B69DF6398D44B17BC6F2 K4SNO = 2 CARDTYPE = SIM ALG = COMP128_3 SN = 8 MSI = 405027002546734 IKVALUE = 1D9F8BAA73973D8FBF8CBFB01436D822 K4SNO = 2 CARDTYPE = SIM ALG =... (8 Replies)
Discussion started by: prasanth_babu
8 Replies

7. Shell Programming and Scripting

Print duplicate only lines as normal output - Awk

input output a1 100 200 XYZ_X a1 98 188 ABC (2 Replies)
Discussion started by: quincyjones
2 Replies

8. Shell Programming and Scripting

merge lines into single line based on symbol \t

The symbols are \t and \t\t (note: not tab) If the line starts with \t merge them into a single line upto symbol \t\t \t\t to end and start new line I able to join in a single line but not ending at \t\t and I completely confused help would be appreciated:b::D Input \ta tab XXXXXXXXXX \te... (5 Replies)
Discussion started by: repinementer
5 Replies

9. Shell Programming and Scripting

Help on Merge multi-lines into one single line

Hello, Can anyone let me know how to use Perl script to Merge following multi-lines into one single line... ***** Multi-line***** FILE_Write root OK Tue Jul 01 00:00:00 2008 cl_get_path file descriptor = 1 FILE_Write root OK ... (5 Replies)
Discussion started by: happyday
5 Replies

10. Shell Programming and Scripting

Merge multi-lines into one single line

Hi, Can anyone help me for merge the following multi-line log which beginning with a number and time: into one line. For each line need to delete the return and add a space. Please see the red color line. *****Original Log*****... (4 Replies)
Discussion started by: happyday
4 Replies
Login or Register to Ask a Question