Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk match and merge with 2 files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk match and merge with 2 files
# 1  
Old 11-06-2013
awk match and merge with 2 files
Dear Awk experts!

I have been trying to get a one liner for a match and merge operation, but having difficulties as I'm an awk newb. I always get stuck on the match and merge with 2 files.

I have 2 files as follows:

File 1:
Code:
field 1,field 2,field 3,field 4,field 5,field 6,field 7,field 8,field 9
1,first,yes,blah,1,a@b.com,m,12345,test 1
2,second,yes,this,1,b@c.com,n,22222,test 2
3,third,no,that,1,a@example.com,n,44444,test 4
4,fourth,no,ok,1,b@example.com,f,57657,test 3

File 2: (field separator is ":" in this file)
Code:
field 1:field 2
12345:additional1
44444:additional2
98777:additional3

So field 8 in file1 is matched with field 1 in file2.

If there is a match, then field 2 from file2 is appended to the end of the row in file 1.

So that the output looks like this:
Code:
field 1,field 2,field 3,field 4,field 5,field 6,field 7,field 8,field 9
1,first,yes,blah,1,a@b.com,m,12345,test 1,additional1
2,second,yes,this,1,b@c.com,n,22222,test 2
3,third,no,that,1,a@example.com,n,44444,test 4,additional2
4,fourth,no,ok,1,b@example.com,f,57657,test 3

If someone could provide a breakdown of the awk statement solution, this would also be very helpful for next time!

Thanks!
Last edited by landossa; 11-07-2013 at 10:00 PM.. Reason: Fix errata
# 2  
Old 11-06-2013
First of all I don't see any matching values in field 7. So I guess you want to match field 8 instead of field 7.

Also your final output does not make any sense!
Code:
field 1,field 2,field 3,field 4,field 5,field 6,field 7,field 8,field 9
1,first,yes,blah,1,a@b.com,m,12345,test 1,additional1
2,second,yes,this,1,b@c.com,n,44444,test 2,
3,third,no,that,1,a@example.com,n,76543,test 4
4,fourth,no,ok,1,b@example.com,f,44444,test 3,additional2

How the highlighted value changed from original input?
This User Gave Thanks to Yoda For This Post:
landossa
# 3  
Old 11-06-2013
Thanks for pointing that out, Yoda. My bad - I fixed it up.
# 4  
Old 11-06-2013
An awk approach:
Code:
awk -F'[:,]' '
        NR == FNR {
                if ( NR > 1 )
                        A[$1] = $2
                next
        }
        {
                if ( FNR == 1 )
                {
                        print
                        next
                }
                if ( FNR > 1 && A[$8] )
                        $0 = $0 "," A[$8]
                print
        }
' file2 file1

This User Gave Thanks to Yoda For This Post:
landossa
# 5  
Old 11-07-2013
Code:
awk -F"[:,]" 'NR==FNR{a[$1]=$2;next}
a[$8]{$0=$0","a[$8]}1' File2 File1

# 6  
Old 11-07-2013
There is another inconsistency in the requested output:
Code:
field 1,field 2,field 3,field 4,field 5,field 6,field 7,field 8,field 9
1,first,yes,blah,1,a@b.com,m,12345,test 1,additional1
2,second,yes,this,1,b@c.com,n,22222,test 2,
3,third,no,that,1,a@example.com,n,44444,test 4,additional2
4,fourth,no,ok,1,b@example.com,f,57657,test 3

Note that a comma was added to the end of the line with 22222 in field 8, but no comma was added to the line with 57657 in field 8 (and neither of those values appear in field 1 in File 2).

If there is no match in File 2, should that input line be unchanged or should an empty field be added? And, should a new field have also been added to the header line in the output file?

Note that Yoda's script assumes that unmatched lines should not be changed and does not change the header line, while pravin27's script assumes an empty field should be added and adds a comma (but not a field name) to the header line.
Last edited by Don Cragun; 11-07-2013 at 05:32 AM.. Reason: Note differences between given scripts.
# 7  
Old 11-07-2013
Code:
awk '
NR==FNR { # first file
  a[$1]=$2 # store in array a, hash-indexed by $1
  next # done first file
} # now remaining file
{ # print, append a[$8] if it exists
  print ($8 in a) ? $0 FS a[$8] : $0
}' FS=":" File2 FS="," File1

This User Gave Thanks to MadeInGermany For This Post:
landossa
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to match field between two files and use conditions on match

I am trying to look for $2 of file1 (skipping the header) in $2 of file2 (skipping the header) and if they match and the value in $10 is > 30 and $11 is > 49, then print the line from file1 to a output file. If no match is foung the line is not printed. Both the input and output are tab-delimited.... (3 Replies)
Discussion started by: cmccabe
3 Replies

2. Shell Programming and Scripting

Merge columns from two files using awk

I have two csv files : say a.csv, b.csv a.csv looks like this : property1,property2,100 property3,property4,200 In a.csv, the combination of column1 and column2 will be unique b.csv looks like this property1,property2, 300, t1 property1,property2, 400,t2 property3, property4,800,t1... (2 Replies)
Discussion started by: Lakshmikumari
2 Replies

3. Shell Programming and Scripting

awk merge two files

file1 AAA3:WWW1:DDD1:XXX8:DDD2:XXX9 AAA6:WWW2:FFF1:XXX130:FFF1:XXX104:FFF1:XXX16 AAA7:WWW3:ZZZ1:XXX4:ZZZ2:XXX5:ZZZ3:XXX6:ZZZ4:XXX7file2 XXX8:EEE1:EEE2 XXX9:KKK1:KKK2 XXX130:OOO1:OOO2 XXX104:PPP1:PPP2 XXX16:RRR1:RRR1 XXX4:UUU1:UUU2 XXX5:III1:III2 XXX7:JJJ1:JJJ2Result... (2 Replies)
Discussion started by: vikus
2 Replies

4. Shell Programming and Scripting

merge two files with awk

I have two file like follows. I want to merge them according the first field of file1. The separator of file1 is tab, while the second one is ",". I tried some code, but can't get the results. File1: AABB 6072 28 5922 BBCC 316 147 162 CCDD 907 71 231 File2: CCDD,hTRBV12-4,hTRBJ2-3,319895... (7 Replies)
Discussion started by: xshang
7 Replies

5. Shell Programming and Scripting

AWK to match and merge data from 2 files into 1.

Hello, hopefully this is an easy on for the AWK guru's out there. I'm having some trouble figuring out how to match+merge data in 2 files into 1 single report. I've got my 2 files filtered and delimited, just need to MATCH $3 in file1 to $1 in file2, then put $0 from File1 and $2+$3 from File2... (6 Replies)
Discussion started by: right_coaster
6 Replies

6. Shell Programming and Scripting

Merge files using AWK

I want to merge data from 2nd file to 1st file based on 1st column File1 ==== data1,12,comp1 data1,13,comp2 data3,14,, File2 ==== data1,11,host1,lit data2,11,host2,lit3 data3,11,host3,lit4 Required Ouput (5 Replies)
Discussion started by: greycells
5 Replies

7. Shell Programming and Scripting

Merge 2 csv files with awk

I have 2 files pipe delimted and want to merge them based on a key e.g file 1 123$aaa$yyy$zzz 345$xab$yzy$zyz 456$sss$ttt$foo 799$aaa$ggg$dee file 2 123$hhh 345$ddd 456$xxx 888$zzz so if the key is the first field, and the result should be the common key between file 1 and 2 (6 Replies)
Discussion started by: loloAix
6 Replies

8. Shell Programming and Scripting

merge two files into one file use awk

Hi, guys. I have one question: I have two files: passwd and shadow (the number of records in these files are not equal)the contents of them are below: passwd: ************** ftp:x:24:24: sshd:x:71:65: uucp:x:10:14: brownj:x:5005:1000: sherrys: x :5006:1000: ... ************* ... (2 Replies)
Discussion started by: daikeyang
2 Replies

9. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

10. Shell Programming and Scripting

merge two files using awk

Hi Guys, I wonder whether is possible to merge two files using awk. I have two files one with 7 columns and another one with 9 columns and the first column on both files is identical so will be my key to merge the files. Any ideas.Thanks in advance. Harby. (2 Replies)
Discussion started by: hariza
2 Replies
Login or Register to Ask a Question