Merge two files with common IDs but unequal number of rows


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Merge two files with common IDs but unequal number of rows
# 1  
Old 03-07-2012
Merge two files with common IDs but unequal number of rows

Hi,

I have two files that I would like to merge and think that there should be a solution using awk. The files look something like this:

file 1
IDX1 IDY1
IDX2 IDY2
IDX3 IDY3

file 2
IDY1 dataA data1
IDY2 dataB data2
IDY3 dataC data3

Desired output
IDX1 IDY1 dataA data1
IDX2 IDY2 dataB data2
IDX3 IDY3 dataC data3

My problem is that in file 2, some of the IDY's are repeated, whereas in file 1 they are not. I would like to keep the repeated entries in file 2 and link the second ID name from file 1.

Hope this makes sense. Thanks in advance for the help!
# 2  
Old 03-07-2012
No need for AWK here:
Code:
join -t" " -12 -21 -o1.1,1.2,2.2,2.3 file1 file2

This User Gave Thanks to bartus11 For This Post:
# 3  
Old 03-07-2012
This should match the last of IDY entries in file2
Code:
awk 'NR==FNR{A[$1]=$0;next}$2=A[$2]' file2 file1

This User Gave Thanks to Scrutinizer For This Post:
# 4  
Old 03-07-2012
Quote:
Originally Posted by Scrutinizer
This should match the last of IDY entries in file2
Code:
awk 'NR==FNR{A[$1]=$0;next}$2=A[$2]' file2 file1

Thanks for this. It did join the files but only seemed to find the first entry for IDY in file 2, so file is the length of file 1. File 1 is ~400 lines and file 2 is ~3,000 lines with multiple lines for the same IDY. I'd like to preserve the length of file 2 but add on the extra ID code from file 1.

But thanks for the start!
# 5  
Old 03-07-2012
OK, now I think I understand what you mean (I thought you meant the other way around)
Try:
Code:
awk 'NR==FNR{A[$2]=$0;next}$1=A[$1]' file1 file2


Last edited by Scrutinizer; 03-07-2012 at 04:22 PM..
This User Gave Thanks to Scrutinizer For This Post:
# 6  
Old 03-07-2012
Works beautifully! Thanks so much!

BTW, couldn't get the join command to work. It returned an empty output. I'll look into it more.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Merge selective columns from files based on common key

Hi, I am trying to selectively merge two files based on keys reported in the 1st column. File1: #file1-header1 file1-header2 111 qwe rtz uio 198 asd fgh jkl 165 yxc 789 poi uzt rew 89 lkj File2: #file2-header2 file2-header2 165 ghz nko2 ... (2 Replies)
Discussion started by: dovah
2 Replies

2. Shell Programming and Scripting

Merge multiple files with common header

Hi all, Say i have multiple files x1 x2 x3 x4, all with common header (date, time, year, age),, How can I merge them to one singe file "X" in shell scripting Thanks for your suggestions. (2 Replies)
Discussion started by: msarguru
2 Replies

3. Shell Programming and Scripting

Merge files based on both common and uncommon rows

Hi, I have two files A (2190 rows) and file B (1100 rows). I want to merge the contents of two files based on common field, also I need the unmatched rows from file A file A: ABC XYZ PQR file B: >LMN|chr1:11000-12456: >ABC|chr15:176578-187678: >PQR|chr3:14567-15866: output... (3 Replies)
Discussion started by: Diya123
3 Replies

4. Shell Programming and Scripting

Merging files with common IDs without JOIN

Hi, I am trying to merge information across 2 files. The first file is a "master" file, with all IDS. File 2 contains a subset of IDs of those in File 1. I would like to match up individuals in File 1 and File 2, and add information in File 2 to that of File 1 if they appear. However, if an... (3 Replies)
Discussion started by: hubleo
3 Replies

5. UNIX for Dummies Questions & Answers

Merge rows with common column

Dear all I have big file with two columns A_AA960715 GO:0006952 A_AA960715 GO:0008152 A_AA960715 GO:0016491 A_AA960715 GO:0007165 A_AA960715 GO:0005618 A_AA960716 GO:0006952 A_AA960716 GO:0005618 A_AA960716... (15 Replies)
Discussion started by: AAWT
15 Replies

6. UNIX for Advanced & Expert Users

merge two tab delimited file with exact same number of rows in unix/linux

Hi I have two tab delimited file with different number of columns but same number of rows. I need to combine these two files in such a way that row 1 in file 2 comes adjacent to row 1 in file 1. For example: The content of file1: field1 field2 field3 a1 a2 a3 b1 b2 b3... (2 Replies)
Discussion started by: mary271
2 Replies

7. UNIX for Dummies Questions & Answers

Writing a loop to merge multiple files by common column

I have 100 data files labelled 250.1.txt through 250.100.txt. The second column of the data files partially match (there is about %90 overlap). Each data file has 4 columns. I want the merge all these text files by the matching values in the second column. In the output, the first column should... (1 Reply)
Discussion started by: evelibertine
1 Replies

8. Shell Programming and Scripting

merge two text files of different size on common index

I have two text files. text file 1: ID filePath col1 col2 col3 1 10584588.mol 269.126 190.958 23.237 2 10584549.mol 281.001 200.889 27.7414 3 10584511.mol 408.824 158.316 29.8561 4 10584499.mol 245.632 153.241 25.2815 5 10584459.mol ... (8 Replies)
Discussion started by: LMHmedchem
8 Replies

9. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

10. Shell Programming and Scripting

merge rows based on a common column

Hi guys, Please guide me if you have a solution to this problem. I have tried paste -s but it's not giving the desired output. I have a file with the following content- A123 box1 B345 bat2 C431 my_id A123 service C431 box1 A123 my_id I need two different outputs- OUTPUT1 A123... (6 Replies)
Discussion started by: smriti_shridhar
6 Replies
Login or Register to Ask a Question