AWK to match and merge data from 2 files into 1.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting AWK to match and merge data from 2 files into 1.
# 1  
Old 06-25-2012
AWK to match and merge data from 2 files into 1.

Hello, hopefully this is an easy on for the AWK guru's out there. I'm having some trouble figuring out how to match+merge data in 2 files into 1 single report.

I've got my 2 files filtered and delimited, just need to MATCH $3 in file1 to $1 in file2, then put $0 from File1 and $2+$3 from File2 into File3

File1
Code:
0623@1100@ABCXYZ
0624@1203@ABCXYZ
0624@1345@XYZ123
0624@1458@123456
0625@0900@123ABC
0625@0930@ABCXYZ

File2
Code:
ABCXYZ@APPLICATION@GROUP
XYZ123@APP1@GROUP1
123456@APP45@GROUP34
123ABC@APP2@GROUP2

File3
Code:
0623 1100 ABCXYZ APPLICATION GROUP
0624 1203 ABCXYZ APPLICATION GROUP
0624 1345 XYZ123 APP1 GROUP1
0624 1458 123456 APP45 GROUP34
0625 0900 123ABC APP2 GROUP2
0625 0930 ABCXYZ APPLICATION GROUP

Note that File1 will contain duplicate lines (which I want to show in file3) but file2 will never contain duplicate lines, if that matters.
# 2  
Old 06-25-2012
Hi, try:
Code:
awk -F@ 'NR==FNR{A[$1=$1]=$0; next} $3=A[$3]' file2 file1

# 3  
Old 06-25-2012
Code:
awk -F@ 'NR == FNR {
  $1 = $1
  f2[$1] = $0
  next
  }
!d[$0]++ {
  print $1, $2, f2[$3]
  }' file2 file1

# 4  
Old 06-25-2012
Hi right_coaster,

One more way:
Code:
$ cat script.awk
BEGIN {
        ## A space as Output File Separator.
        OFS = " "

        ## Field Separator.
        FS = "@"
}

## First input file. Save data in an array, field 1 as key and
## fields 2 and 3 as values.
FNR == NR {
        f2[ $1 ] = $2 " " $3
        next
}

## Second input file. If third field is found, do a no-op to 
## use OFS and print.
FNR < NR {
        if ( f2[ $3 ] ) {
                $1 = $1
                print $0, f2[ $3 ]
        }
}
$ awk -f script.awk file2 file1
0623 1100 ABCXYZ APPLICATION GROUP
0624 1203 ABCXYZ APPLICATION GROUP
0624 1345 XYZ123 APP1 GROUP1
0624 1458 123456 APP45 GROUP34
0625 0900 123ABC APP2 GROUP2
0625 0930 ABCXYZ APPLICATION GROUP

# 5  
Old 06-26-2012
Not working on my actual files

Thanks for the replies, they worked for my small example in the post. I tested it on a couple of test files. However when I applied it to my actual files, nothing is returned from the awk command.

I verified that they are formated exactly like the example, there is no whitespace in the files, etc. I tried every reply given here, all simply return me to the prompt with no output. I also tried swapping the order of file1 file2 in the command (just to make sure), and nothing.

Any ideas, i'm baffled? I've attached the real files, apps2.txt is file2 and notok.txt is file1.
# 6  
Old 06-26-2012
Nevermind my last post

I did find additional white space in my file which I removed with
Code:
tr -d ' '

Thanks again for the replies! I'm all set.
# 7  
Old 06-27-2012
Try this ,
Code:
 awk -F"@" 'NR==FNR{a[$1]=$2 " " $3; next}
{gsub(" ","",$3); if(a[$3]) { print $0 , a[$3]}}' apps2.txt notok.txt

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Data match 2 files based on first 2 columns matching only and join if match

Hi, i have 2 files , the data i need to match is in masterfile and i need to pull out column 3 from master if column 1 and 2 match and output entire row to new file I have tried with join and awk and i keep getting blank outputs or same file is there an easier way than what i am... (4 Replies)
Discussion started by: axis88
4 Replies

2. Shell Programming and Scripting

awk to match field between two files and use conditions on match

I am trying to look for $2 of file1 (skipping the header) in $2 of file2 (skipping the header) and if they match and the value in $10 is > 30 and $11 is > 49, then print the line from file1 to a output file. If no match is foung the line is not printed. Both the input and output are tab-delimited.... (3 Replies)
Discussion started by: cmccabe
3 Replies

3. Shell Programming and Scripting

Merge and Sort tabular data from different text files

I have 42 text files; each containing up to 34 lines with following structure; file1 H-01 23 H-03 5 H-05 9 H-02 14 . . file2 H-01 17 H-02 43 H-04 7 H-05 8 H-03 7 . . file3 (6 Replies)
Discussion started by: Syeda Sumayya
6 Replies

4. Shell Programming and Scripting

Merge files and copy some data using sed or awk

Copy data from other file to paste cat file1: Name: server1.data.com data1 server1 running Name: server3.data.com data3 server3 running cat file2: server1 good server2 bad network not ok server3 good Output: (10 Replies)
Discussion started by: kenshinhimura
10 Replies

5. Shell Programming and Scripting

How to merge the multiple data files as a single file?

Hi Experts, I have created multiple scripts and send the output to new file, getting this output to my mailbox on daily basis. I would like to send the all outputs to a single file, need to merge all file outputs on a single file. For example, Created script for df -h > df.doc grep... (7 Replies)
Discussion started by: seenuvasan1985
7 Replies

6. Shell Programming and Scripting

awk match and merge with 2 files

Dear Awk experts! I have been trying to get a one liner for a match and merge operation, but having difficulties as I'm an awk newb. I always get stuck on the match and merge with 2 files. I have 2 files as follows: File 1: field 1,field 2,field 3,field 4,field 5,field 6,field 7,field... (6 Replies)
Discussion started by: landossa
6 Replies

7. Shell Programming and Scripting

awk help: Match data fields from 2 files & output results from both into 1 file

I need to take 2 input files and create 1 output based on matches from each file. I am looking to match field #1 in both files (Userid) and create an output file that will be a combination of fields from both file1 and file2 if there are any differences in the fields 2,3,4,5,or 6. Below is an... (5 Replies)
Discussion started by: ambroze
5 Replies

8. Shell Programming and Scripting

Merge two file data together based on specific pattern match

My input: File_1: 2000_t g1110.b1 abb.1 2001_t g1111.b1 abb.2 abb.2 g1112.b1 abb.3 2002_t . . File_2: 2000_t Ali england 135 abb.1 Zoe british 150 2001_t Ali england 305 g1111.b1 Lucy russia 126 (6 Replies)
Discussion started by: patrick87
6 Replies

9. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

10. Shell Programming and Scripting

Merge 70 files into one data matrix

Hi, I have a list of 70 files in a directory and I need to merge the content of each file into one big matrix file (71 columns x 3060 rows). Each file has the following format only two columns per file: unique identifier1 randomtext1 randomtext1 a 5 b 3 c 6 d 3 e 2... (11 Replies)
Discussion started by: labrazil
11 Replies
Login or Register to Ask a Question