Merging columns based on one or more column in two files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merging columns based on one or more column in two files
# 1  
Old 08-30-2012
Merging columns based on one or more column in two files

I have two files.
FileA.txt
Code:
30910   rs7468327
36587   rs10814410
91857   rs9408752
105797  rs1133715
146659  rs2262038
152695  rs2810979
181843  rs3008128
182129  rs3008131
192118  rs3008170

FileB.txt
Code:
30910 1.9415219673 0
36431 1.3351312477 0.0107191428
36587 1.3169171182 0.0109274233
37123 1.3181466012 0.0116332908
38515 1.1211025231 0.0134681509
44551 1.5498135416 0.0202351257
47327 1.5694610726 0.0245374081
48265 1.5556343019 0.0260095626
68775 1.5538580867 0.0579156221

I want to merge the columns together based on Column 1.
Also would like to know if I can merge these together if I had more than one column to match between the two files.
# 2  
Old 08-30-2012
If file one isn't too large, then this should work

Code:
# single pass across each file, but requires the entire first file
# to be held in memory which might not be realistic.
# order is preserved based on file2
awk '
    NR == FNR { cache[$1] = $0; next; }
    $1 in cache {
        printf( "%s", cache[$1] );
        $1 = "";
        print;
    }
' file1 file2 >output

If file1 is large (i.e. it's not practical to cache it in memory), then this is one way. May not be the most efficent, but it should work. The order of the output is sorted by field1.

Code:
# multiple passes across the data, but memory requirement is eliminated
# order of file2 is not preserved.
(
    sed 's/^/a /' file1
    sed 's/^/b /' file2
) | sort -k 2n,2 -k 1,1  awk '
    $1 == "a" {
        x = $2;
        $1 = "";
        cache = $0;
        next;
    }
    $2 == x {
        $1 = $2 ="";
        printf( "%s%s\n", substr( cache, 2 ), $0 );
    }
'

You could do this without the seds, and depend on the number of columns to determine if an unmatched pair exists, but this works without having to know the exact layout of either file, other than the desired column to compare.

Yes, multiple columns can be used to match.

Last edited by agama; 08-30-2012 at 11:38 PM.. Reason: small efficiency change.
# 3  
Old 08-31-2012
The title leads me to join:

Code:
join FileA.txt FileB.txt

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merging rows based on same ID in First column.

Hellow, I have a tab-delimited file with 3 columns : BINPACKER.13259.1.p2 SSF48239 BINPACKER.13259.1.p2 PF13243 BINPACKER.13259.1.p2 G3DSA:1.50.10.20 BINPACKER.13259.2.p2 SSF48239 BINPACKER.13259.2.p2 PF13243 BINPACKER.13259.2.p2 G3DSA:1.50.10.20... (7 Replies)
Discussion started by: anjaliANJALI
7 Replies

2. Shell Programming and Scripting

Paste columns based on common column: multiple files

Hi all, I've multiple files. In this case 5. Space separated columns. Each file has 12 columns. Each file has 300-400K lines. I want to get the output such that if a value in column 2 is present in all the files then get all the columns of that value and print it side by side. Desired output... (15 Replies)
Discussion started by: genome
15 Replies

3. UNIX for Dummies Questions & Answers

Merging two files based on matching columns

Hi, I am facing issues while accomplishing below task. We have two files Test1.txt and Test2.txt. We have to match 1st column of Test1.txt file with 2nd column of Test2.txt and then merge 2nd file with the 1st file. In the output we should select column 1 and 2 from the 1st file and column 1... (5 Replies)
Discussion started by: Prathmesh
5 Replies

4. Shell Programming and Scripting

Merging two file based on comparison of first columns

Respected Members. Hello. This is my first post in the forum. I will try to follow all the rules as prescribed by the forum. In case of non-compliance, I request you to kindly give me some more time to understand and abide by them. I am working on two files. I wish to merge the two files... (6 Replies)
Discussion started by: manojmalhotra
6 Replies

5. Shell Programming and Scripting

Merging two file based on comparison of first columns

Respected Members. Hello. This is my first post in the forum. I will try to follow all the rules as prescribed by the forum. In case of non-compliance, I request you to kindly give me some more time to understand and abide by them. I am working on two files. I wish to merge the two files... (1 Reply)
Discussion started by: manojmalhotra
1 Replies

6. UNIX for Dummies Questions & Answers

Merging lines based on one column

Hi, I have a file which I'd like to merge lines based on duplicates in one column while keeping the info for other columns. Let me simplify it by an example: File ESR1 ANASTROZOLE NA FDA_approved ESR1 CISPLATIN NA FDA_approved ESR1 DANAZOL agonist NA ESR1 EXEMESTANE NA FDA_approved... (3 Replies)
Discussion started by: JJ001
3 Replies

7. Shell Programming and Scripting

merging two files based on first column

I had two files file1 and file2. I want a o/p file(file3) like below using first column as ref. Pls give suggestion ass join is not working as the number of lines in each file is nealry 5 C? file1 --------------------- 404000324810001 Y 404000324810004 N 404000324810008 Y 404000324810009 N... (1 Reply)
Discussion started by: p_sai_ias
1 Replies

8. UNIX for Dummies Questions & Answers

Merging two files based on two columns to make a third file

Hi there, I'm trying to merge two files and make a third file. However, two of the columns need to match exactly in both files AND I want everything from both files in the output if the two columns match in that row. First file looks like this: chr1 10001980 T A Second... (12 Replies)
Discussion started by: infiniteabyss
12 Replies

9. Shell Programming and Scripting

Merging 2 files based on a common column

Hi All, I do have 2 files file 1 has 4 tab delimited columns 234 a c dfgyu 294 b g fih 302 c h jzh 328 z c san 597 f g son File 2 has 2 tab delimted columns 234 23 302 24 597 24 I want to merge file 2 with file 1 based on the data common in both files which is the first column so... (6 Replies)
Discussion started by: Lucky Ali
6 Replies

10. Shell Programming and Scripting

merging column from two files based on identifier

Hi, I have two files consisting of two columns. So I want to merge column 2 if column 1 is the same. So heres an example of what I mean. FILE1 driver 444 car 333 hat 222 FILE2 driver 333 car 666 hat 999 So I want to merge the column 2's together so... (4 Replies)
Discussion started by: phil_heath
4 Replies
Login or Register to Ask a Question