AWK Matching Fields and Combining Files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting AWK Matching Fields and Combining Files
# 1  
Old 03-29-2009
AWK Matching Fields and Combining Files

Hello!

I am writing a program to run through two large lists of data (~300,000 rows), find where rows in one file match another, and combine them based on matching fields. Due to the large file sizes, I'm guessing AWK will be the most efficient way to do this. Overall, the input and output I'm looking for is similar to to this:

File1: *first three columns are coordinates in (x, y, z)*
123 456 678 A B C
234 345 567 D F B
234 456 324 H J K
765 432 987 M N K


File2: *the last three columns are coordinates in (x, y, z)*
45 234 345 567
46 765 432 987
47 111 222 333
48 234 345 567
49 987 765 432
50 444 555 666
51 765 432 987
... and so on

Output file:
45 234 345 567 D F B
46 765 432 987 M N K
48 234 345 567 D F B
51 765 432 987 M N K

File2 has many more entries than File1, and every coordinate in File1 is located somewhere in File2. The problem I am having is how to search through all of File2 finding where each of the individual File1 coordinates is listed, and the number in column 1 of File2 that corresponds to that coordinate.

In a nutshell:
Make new file3
Find where File2($2, $3, $4) is equal to File1($1, $2, $3)
print to file3 File2($1, $2, $3, $4), File1($4, $5, $6)

Thank you!
# 2  
Old 03-29-2009
Use gawk, nawk or /usr/xpg4/bin/awk on Solaris.

Code:
awk > file3 'NR == FNR {
  _[$1, $2, $3] = $4 FS $5 FS $6
  next
  }
($2, $3, $4) in _ {
  print $0, _[$2, $3, $4]
  }' file1 file2

If you're really sure that for every key in file2 there is an entry in file1
you can remove the ($2, $3, $4) in _ conditional expression.

Last edited by radoulov; 03-30-2009 at 04:40 AM..
# 3  
Old 03-29-2009
just another way

was bit late than radoulovSmilie
Code:
awk 'FILENAME=="file1"{a[$1$2$3]=$4" "$5" "$6}
FILENAME=="file2"{if(a[$2$3$4]){print $0" "a[$2$3$4]}}' file1 file2 > file3

# 4  
Old 03-29-2009
First create a sorted copy of file1 and file2 then use join to create your output file3
Code:
sort -nk 1,1 file1 > file1_sorted
sort -nk 2,2 file2 > file2_sorted

join -t" " -1 1 -2 2 -o 2.1,2.2,2.3,2.4,1.4,1.5,1.6 file1_sorted file2_sorted > file3


Last edited by ldapswandog; 03-30-2009 at 10:26 PM..
# 5  
Old 03-29-2009
Bug Thank you! Thank you! Thank you!

Thank you very much for all the help. I'm a first-timer on the UNIX/LINUX forums, and definitely plan to come back when/if (okay, let's be honest - when) I need help again.

What fast replies, I was sure I'd have to wait until Monday for a response. I've tried all three of your suggestions and they all work beautifully!

Thank you again and enjoy your Sunday (or what's left of it)!
# 6  
Old 03-30-2009
Just a suggestion if you have to perform this function often test each solution using 'time' to see which one is more efficient.
Code:
sort -nk 1,1 file1 > file1_sorted
sort -nk 2,2 file2 > file2_sorted

time join -t" " -1 1 -2 2 -o 2.1,2.2,2.3,2.4,1.4,1.5,1.6 file1_sorted file2_sorted > file3

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk for matching fields between files with repeated records

Hello all, I am having trouble with what should be an easy task, but seem to be missing something fundamental. I have two files, with File 1 consisting of a single field of many thousands of records. I also have File 2 with two fields and many thousands of records. My goal is that when $1 of... (2 Replies)
Discussion started by: jvoot
2 Replies

2. UNIX for Beginners Questions & Answers

Continued trouble matching fields in different files and selective field printing ([g]awk)

I apologize in advance, but I continue to have trouble searching for matches between two files and then printing portions of each to output in awk and would very much appreciate some help. I have data as follows: File1 PS012,002 PRQ 0 1 1 17 1 0 -1 3 2 1 2 -1 ... (7 Replies)
Discussion started by: jvoot
7 Replies

3. Shell Programming and Scripting

awk to print fields that match using conditions and a default value for non-matching in two files

Trying to use awk to match the contents of each line in file1 with $5 in file2. Both files are tab-delimited and there may be a space or special character in the name being matched in file2, for example in file1 the name is BRCA1 but in file2 the name is BRCA 1 or in file1 name is BCR but in file2... (6 Replies)
Discussion started by: cmccabe
6 Replies

4. UNIX for Beginners Questions & Answers

Awk: matching multiple fields between 2 files

Hi, I have 2 tab-delimited input files as follows. file1.tab: green A apple red B apple file2.tab: apple - A;Z Objective: Return $1 of file1 if, . $1 of file2 matches $3 of file1 and, . any single element (separated by ";") in $3 of file2 is present in $2 of file1 In order to... (3 Replies)
Discussion started by: beca123456
3 Replies

5. Shell Programming and Scripting

Awk: adding fields after matching $1

Dear AWK-experts! I did get stuck in the task of combining files after matching fields, so I'm still awkward with learning AWK. There are 2 files: one containing 3 columns with ID, coding status, and score for long noncoding RNAs: file1 (1.txt) (>5000 lines) ... (12 Replies)
Discussion started by: kben
12 Replies

6. Shell Programming and Scripting

AWK- delimiting the strings and matching the fields

Hello, I am newbie in awk. I have just started learning it. 1) I have input file which looks like: {4812 4009 1602 2756 306} {4814 4010 1603 2757 309} {8116 9362 10779 } {10779 10121 9193 10963 10908} {1602 2756 306 957 1025} {1603 2757 307} and so on..... 2) In output: a)... (10 Replies)
Discussion started by: kajolo
10 Replies

7. Shell Programming and Scripting

To get an output by combining fields from two different files

Hi guys, I couldn't find solution to this problem. If anyone knows please help me out. your guidance is highly appretiated. I have two files - FILE1 has the following 7 columns ( - has been added to make columns visible enough else columns are separated by single space) 155.34 - leg - 1... (8 Replies)
Discussion started by: smriti_shridhar
8 Replies

8. Shell Programming and Scripting

Matching and combining two files

Hi there, I have two files. What I want to do is search for the values in second field of file1 in the 6th field of the file2 and of they match to add the fields 1-5 of the file2 at the end of the line of file1 with a comma before. E.g File1 FWB,CHUAGT87HUMAS/BUD01,REUAIR08KLM... (3 Replies)
Discussion started by: sickboy
3 Replies

9. Shell Programming and Scripting

combining fields in awk

I am using: ps -A -o command,%cpu to get process and cpu usage figures. I want to use awk to split up the columns it returns. If I use: awk '{print "Process: "$1"\nCPU Usage: "$NF"\n"}' the $NF will get me the value in the last column, but if there is more than one word in the... (2 Replies)
Discussion started by: json4639
2 Replies

10. Shell Programming and Scripting

Matching and combining two files

Hi, How can I match the first two fields of file2 against the first two fields of file1 and where they match combine the two lines. If the name (example-Aidan Rielly) is in file1 but not in file2 then just write the info from file1 to the combined output file. If the name (example-Silvia... (5 Replies)
Discussion started by: p3t3r
5 Replies
Login or Register to Ask a Question