Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

compare two files and to remove the matching lines on both the files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting compare two files and to remove the matching lines on both the files
# 1  
Old 06-19-2009
compare two files and to remove the matching lines on both the files
I have two files and need to compare the two files and to remove the matching lines from both the files
# 2  
Old 06-19-2009
Code:
awk '{arr[$0]++} END {for (i in arr) {if (arr[i]>1) {print $0} }} ' file1 file2 > dupfile
awk 'FILENAME=="dupfile" {arr[$0]++}
       FILENAME=="file1" {if ($0 in arr) {next} else {print $0 >"newfile1"}}
       FILENAME=="file2" {if ($0 in arr) {next} else {print $0 >"newfile2"} }' dupfile file1 file2

Not tested.
# 3  
Old 06-19-2009
Try this...

Code:
 
awk '{ if (FNR==NR) {arr[$0]++;next}
if ($0 in arr) { arr[$0]--; if (arr[$0] == 0) delete arr[$0]; next}
{print $0 >"newfile2"} } 
END {for (i in arr) {print i >"newfile1"} } ' file1 file2

Jim: The code you have given has potential to remove duplicate lines in one file even though no matching record present in other file because those lines has multiple instances in a single file.
Last edited by King Kalyan; 06-19-2009 at 05:06 PM.. Reason: Corrected the code for duplicates also...
# 4  
Old 06-20-2009
I also have this kind of problem. The code above deletes the lines which are alike on the two files. How can i do this, on a separate file?
File 3:
#from first file
a3px lm4 xpm
0wmp jga1 wmp6

#from second file
a3px lm4 xpj
0xmp lm4 xpm

thanks a lot!
# 5  
Old 06-20-2009
maybe?

Code:
awk '! LINE[$0] { print; LINE[$0] = $0}' in_file

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare two files and count number of matching lines

Dear All, I would like to compare two files and return the number of matches found. Example File A Lx2 L1_Mus1 L1Md_T Lx5 L1M2 L1_Mus3 Lx3_Mus Lx9 Lx2A L1Md_A L1Md_F2 File B L1_Mus3 L1_Mus3 (3 Replies)
Discussion started by: paolo.kunder
3 Replies

2. Linux

Remove matching files from a list

hi, i have a cache file with below file list more gtda_11.cache GTDA_Dly_Pmix_GB_6_20130624.20130624070610.psv GTDA_Dly_Pmix_CH_006_20130624.20130624140018.psv GTDA_Dly_Pmix_GB_6_20130624.20130624070620.psv GTDA_Dly_Pmix_BE_6_20130624.20130624070620.psv... (2 Replies)
Discussion started by: renuk
2 Replies

3. Shell Programming and Scripting

Compare and matching column entries in 2 files and

I have 2 files. File 1 has more columns (6 columns but the last column has spaces) than file 2 (file 2 has 4 columns). The entries in file 1 do not change but column 4 in file 2 can be different from the the entry in file 1. I want to create a script that reads in file 1 and then uses column 1 2... (5 Replies)
Discussion started by: kieranfoley
5 Replies

4. Shell Programming and Scripting

Compare file1 for matching line in file2 and print the difference in matching lines

Hello, I have two files file 1 and file 2 each having result of a query on certain database tables and need to compare for Col1 in file1 with Col3 in file2, compare Col2 with Col4 and output the value of Col1 from File1 which is a) not present in Col3 of File2 b) value of Col2 is different from... (2 Replies)
Discussion started by: RasB15
2 Replies

5. Shell Programming and Scripting

Compare one files with strings from another + remove lines

Have two files and want to compare the content of file1 with file2. When matched remove the line. awk 'NR==FNR {b; next} !(b in $0)' file1 file2file1 1. if match 2. removefile2 1. this line has to be removed if match 2. this line has a match, remove 3. this line has no match, no removingThe... (3 Replies)
Discussion started by: sdf
3 Replies

6. Shell Programming and Scripting

Compare two files and get matching data

Hi, Can anyone help me to compare two files and get the matching data... say i have file1 and file2 ... file1 has 300 unique data with that i need to match with file2 to see how may are matching.. file2 have 1000 records. (4 Replies)
Discussion started by: zooby
4 Replies

7. Shell Programming and Scripting

compare files and then remove some lines

Hi everyone I have a dilemma and I'm hoping someone has an answer for me. I have two files: # cat masterfile line3 line4 line5 line6 line7 # cat tempfile line1 line2 line3 line4 I want to compare tempfile with masterfile. (3 Replies)
Discussion started by: soliberus
3 Replies

8. Shell Programming and Scripting

compare two columns of different files and print the matching second file..

Hi, I have two tab separated files; file1: S.No ddi fi cu o/l t+ t- 1 0.5 0.6 o 0.1 0.2 2 0.2 0.3 l 0.3 0.4 3 0.5 0.8 l 0.1 0.6 ... (5 Replies)
Discussion started by: vasanth.vadalur
5 Replies

9. Shell Programming and Scripting

compare two files for matching in solaris

I am using solaris, need either awk/shell or perl script to compate two files. both of these file1 and file2 are located in two diffent location path in the same server. file1 location: /export/db/mna file2: /etc/dba The script will compare in file1 first field (e.g. abc00asp)before colon:... (8 Replies)
Discussion started by: amir07
8 Replies

10. UNIX for Dummies Questions & Answers

Find matching lines between 2 files

How do I find matching lines between two files? (5 Replies)
Discussion started by: jojojmac5
5 Replies
Login or Register to Ask a Question