compare two files based on common field in unix


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers compare two files based on common field in unix
# 1  
Old 12-06-2011
compare two files based on common field in unix

I have two files in UNIX.
1st file is Entity and Second File is References. 1st File has only one column named Entity ID and 2nd file has two columns Entity ID | Person ID.
I want to produce a output file where entity id's are matching in both the files.
Entity File
Code:
624197
624252
624264
624276
624280
624309
624317

Reference File
Code:
624252|624346
624264|1070122
624264|624346
624276|624588
624280|624346
624280|624582
624298|624588
624319|333008
624330|624588

Output File
Code:
624252|624346
624264|1070122
624264|624346
624276|624588
624280|624346
624280|624582

Entity Files has 90K Records and Reference file has 200K Records. Is there an efficient way to produce the third file ?
Any solution is appreciated.

Moderator's Comments:
Mod Comment How to use code tags

Last edited by Franklin52; 12-07-2011 at 05:26 AM.. Reason: Please use code tags for code and data samples, thank you
# 2  
Old 12-06-2011
Try:
Code:
join -t"|" -j1 -o2.1,2.2 Entity Reference

# 3  
Old 12-06-2011
I tried this but the code doesn't produce the desired results. It comes out with 230 rows only. But I know there are already more than 20000 rows. Please help.

---------- Post updated at 05:41 PM ---------- Previous update was at 05:29 PM ----------

Somehow join command has some limitations and it's not producing all the rows as an output.

---------- Post updated at 09:14 PM ---------- Previous update was at 05:41 PM ----------

Any help in resolving this is appreciated.
# 4  
Old 12-06-2011
try with awk..
Code:
awk -F"|" 'NR==FNR{a[$0];next} ($1 in a) {print $0} ' Entity Reference

Note: if you are using join, you must use sorted input.

Last edited by Franklin52; 12-07-2011 at 05:27 AM.. Reason: Please use code tags for code and data samples, thank you
# 5  
Old 12-07-2011
I think there is a syntax error.
Code:
$>awk -F"|" 'NR==FNR{a[$0];next} ($1 in a) {print $0} ' Entity Reference
awk: syntax error near line 1
awk: bailing out near line 1

I have tired removing the space. I am trying to execute in KSH. Any help in resolving this is appreciated.

Also join command I tried before on the sorted fiiles. But it seems to have some limitation and it is comming out after some records.

Moderator's Comments:
Mod Comment Please use code tags!


---------- Post updated at 12:27 PM ---------- Previous update was at 08:29 AM ----------

any help here?

Last edited by zaxxon; 12-07-2011 at 09:55 AM.. Reason: code tags, see PM
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Compare two files and print based on common variable value.

Hi All, i have below two files. FILE: NAME="/dev/sda" TYPE="disk" SIZE="60G" OWNER="root" GROUP="disk" MODE="brw-rw----" PKNAME="" MOUNTPOINT="" NAME="/dev/sda1" TYPE="part" SIZE="500M" OWNER="root" GROUP="disk" MODE="brw-rw----" PKNAME="/dev/sda" MOUNTPOINT="/boot" NAME="/dev/sda2"... (3 Replies)
Discussion started by: balu1234
3 Replies

2. Shell Programming and Scripting

Compare multiple files, and extract items that are common to ALL files only

I have this code awk 'NR==FNR{a=$1;next} a' file1 file2 which does what I need it to do, but for only two files. I want to make it so that I can have multiple files (for example 30) and the code will return only the items that are in every single one of those files and ignore the ones... (7 Replies)
Discussion started by: castrojc
7 Replies

3. Shell Programming and Scripting

Compare two files Field by field and output the result in another file

Hi Friends, Need Help. I have file1.txt as File1.txt |123|A|7267|Hyder|Cross|Sell|7801 |995|A|7051|2008|Lunar|New|Year|Promotion|7801 |996|A|7022|Q108|Targ|Prospect|&|SSCC|Savings|Promo|7801 |997|A|7182|Q1|Feb-Apr|08|Credit|ITA|PA|SBA|Campaign|7801 File2.txt... (7 Replies)
Discussion started by: i150371485
7 Replies

4. Shell Programming and Scripting

Compare a common field in two files and append a column from File 1 in File2

Hi Friends, I am new to Shell Scripting and need your help in the below situation. - I have two files (File 1 and File 2) and the contents of the files are mentioned below. - "Application handle" is the common field in both the files. (NOTE :- PLEASE REFER TO THE ATTACHMENT "Compare files... (2 Replies)
Discussion started by: Santoshbn
2 Replies

5. Shell Programming and Scripting

Matching and Merging csv data fields based on a common field

Dear List, I have a file of csv data which has a different line per compliance check per host. I do not want any omissions from this csv data file which looks like this: date,hostname,status,color,check 02-03-2012,COMP1,FAIL,Yellow,auth_pass_change... (3 Replies)
Discussion started by: landossa
3 Replies

6. Shell Programming and Scripting

Merging CSV fields based on a common field

Hi List, I have two files. File1 contains all of the data I require to be processed, and I need to add another field to this data by matching a common field in File2 and appending a corresponding field to the data in File1 based on the match... So: File 1:... (1 Reply)
Discussion started by: landossa
1 Replies

7. Shell Programming and Scripting

AWK: Pattern match between 2 files, then compare a field in file1 as > or < field in file2

First, thanks for the help in previous posts... couldn't have gotten where I am now without it! So here is what I have, I use AWK to match $1 and $2 as 1 string in file1 to $1 and $2 as 1 string in file2. Now I'm wondering if I can extend this AWK command to incorporate the following: If $1... (4 Replies)
Discussion started by: right_coaster
4 Replies

8. Shell Programming and Scripting

join files based on a common field

Hi experts, Would you please help me with this? I have several files and I need to join the forth field of them based on the common first field. here's an example... first file: 280346 39.88 -75.08 547.8 280690 39.23 -74.83 538.7 280729 40.83 -75.08 499.2 280907 40.9 -74.4 507.8... (5 Replies)
Discussion started by: GoldenFire
5 Replies

9. Shell Programming and Scripting

How to append two files with common field.

I have two files like File1 : will get this file from "who" command. It is a unix file. user val1 Jul 29 13:15 (IP Address1) user val3 Jul 30 03:21 (IP Address2) user val2 Jul 29 13:16 (IP Address3) user val4 Jul 29 13:17 (IP Address4) ... (4 Replies)
Discussion started by: manneni prakash
4 Replies

10. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies
Login or Register to Ask a Question