Compare two files and output difference, by first field using awk.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Compare two files and output difference, by first field using awk.
# 1  
Old 11-03-2011
Compare two files and output difference, by first field using awk.

It seems like a common task, but I haven't been able to find the solution.

vitallog.txt
Code:
1310,John,Hancock
13211,Steven,Mills
122,Jane,Doe
138,Thoms,Doe
1500,Micheal,May

vitalinfo.txt
Code:
12122,Jane,Thomas
122,Janes,Does
123,Paul,Kite

**OUTPUT**
vitalfiltered.txt
Code:
12122,Jane,Thomas
123,Paul,Kite

I tried for two days trying. I tried sort, uniq, grep, etc..., and it seems the answer lies with awk.

Thank You.

Last edited by charles33; 11-03-2011 at 08:10 PM..
# 2  
Old 11-03-2011
Code:
grep -vf vitallog.txt vitalinfo.txt > vitalfiltered.txt

Code:
awk -F, 'NR==FNR{_1[$1]++;next}!_1[$1]' vitallog.txt vitalinfo.txt > vitalfiltered.txt

--ahamed
This User Gave Thanks to ahamed101 For This Post:
# 3  
Old 11-04-2011
Amazing

First off I want to thank you, ahamed101. Thanks!

Ok, I was skeptical at first. I tried the awk solution, on the sample above; it worked. Still skeptical, I tried it on a few large "test" data sets; it worked. I'm still skeptical, before trying on large "production" data sets. So I will keep testing and report back.

I am completely blown away, so far. Which means something has got to be missing; to good to be true!

If this works, I will make ahamed101, my COD WAW nickname Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Simple awk command to compare two files and print first difference

Hello, I have two text files, each with a single column, file 1: 124152970 123899868 123476854 54258288 123117283 file 2: 124152970 123899868 54258288 123117283 122108330 (5 Replies)
Discussion started by: LMHmedchem
5 Replies

2. Shell Programming and Scripting

Compare two CSV files and put the difference in third file with line no,field no and diff value.

I am having two csv files i need to compare these files and the output file should have the information of the differences at the field level. For Example, File 1: A,B,C,D,E,F 1,2,3,4,5,6 File 2: A,C,B,D,E,F 1,2,4,5,5,6 out put file: (12 Replies)
Discussion started by: karingulanagara
12 Replies

3. Shell Programming and Scripting

Script to compare 2 files and prints difference as output sidebyside

Hi All, Am trying script to compare 2 files and print the difference found from old file to new file on line by line basis on side by side display. Basically line by line comparision and files may contain blank line as well I know we have compare/diff commands but i don't how to make... (10 Replies)
Discussion started by: Optimus81
10 Replies

4. Shell Programming and Scripting

Compare two text files and output difference

Hi experts, I am trying to compare two text files and output the difference to another file. I'm not strictly looking for differences in text but additional text at the end of one file that isn't in another, so basically comparing the file 2 against file 1 and printing any additional text to... (9 Replies)
Discussion started by: martin0852
9 Replies

5. Shell Programming and Scripting

Plz Help. Compare 2 files field by field and get the output in another file.

Hi Freinds, I have 2 files . one is source.txt and second one is target.txt. I want to keep source.txt as baseline and compare target.txt. please find the data in 2 files and Expected output. Source.txt 1|HYD|NAG|TRA|34.5|1234 2|CHE|ESW|DES|36.5|134 3|BAN|MEH|TRA|33.5|234... (5 Replies)
Discussion started by: i150371485
5 Replies

6. Shell Programming and Scripting

Compare two files Field by field and output the result in another file

Hi Friends, Need Help. I have file1.txt as File1.txt |123|A|7267|Hyder|Cross|Sell|7801 |995|A|7051|2008|Lunar|New|Year|Promotion|7801 |996|A|7022|Q108|Targ|Prospect|&|SSCC|Savings|Promo|7801 |997|A|7182|Q1|Feb-Apr|08|Credit|ITA|PA|SBA|Campaign|7801 File2.txt... (7 Replies)
Discussion started by: i150371485
7 Replies

7. Shell Programming and Scripting

AWK: Pattern match between 2 files, then compare a field in file1 as > or < field in file2

First, thanks for the help in previous posts... couldn't have gotten where I am now without it! So here is what I have, I use AWK to match $1 and $2 as 1 string in file1 to $1 and $2 as 1 string in file2. Now I'm wondering if I can extend this AWK command to incorporate the following: If $1... (4 Replies)
Discussion started by: right_coaster
4 Replies

8. Shell Programming and Scripting

AWK Compare files, different fields, output

Hi All, Looking for a quick AWK script to output some differences between two files. FILE1 device1 1.1.1.1 PINGS device1 2.2.2.2 PINGS FILE2 2862 SITE1 device1-prod 1.1.1.1 icmp - 0 ... (4 Replies)
Discussion started by: stacky69
4 Replies

9. Shell Programming and Scripting

AWK to compare two files for each field value

I have "n" files in directory A and "n" files in directory B. The files are expected to be the same with same data. Each file has 14 columns and "x" rows. Of the 14 column, 2 columns are to be considered as key identifiers. Based on this unique combination, I need to compare each field value... (2 Replies)
Discussion started by: Sangtha
2 Replies

10. UNIX for Dummies Questions & Answers

Compare Files and Output Difference

I have to compare two files for any differences, then output the lab and question number for any differences. This is what I currently have: diff lab2.txt lab2answer.txt > lab2compare.txt Though the output doesn't have to be sent to a .txt (or any sort of log), I found that easier, at least... (2 Replies)
Discussion started by: Joesgrrrl
2 Replies
Login or Register to Ask a Question