Deleting lines of a file if they exist in another file

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Deleting lines of a file if they exist in another file
# 1  
Old 08-29-2011
Deleting lines of a file if they exist in another file

I have a reference file that needs to remain static and another file that may or may not have duplicate rows that match the reference file. I need help with a command that will delete any duplicate rows from the second file while leaving reference file intact

For example reference file would have a list of colors -

And the second file has the follwing list -

I would like to see the second file end up with only unique rows -

any help with a statement would be greatly appreciated. Thanks
# 2  
Old 08-29-2011
Is this what you are looking for:
sort -u Second_File

# 3  
Old 08-29-2011
awk 'FILENAME="reference" {arr [$0]++}
       FILENAME=="second_file" { if( ! $0 in arr) {print $0} } 
      '   reference   second_file  | sort -u

You want a unique list of values in second file that are NOT in the reference file, coorect?
# 4  
Old 08-29-2011
If you want unique records from the second file that are not in the reference file:
egrep -v -f Reference_File Second_File | sort -u

This User Gave Thanks to Shell_Life For This Post:
# 5  
Old 08-29-2011
Thanks all for your very timely responses! The last post was the first I tried and it worked beautifully!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Deleting the lines exist between the Mattched Patterns

Hi, I have a file with the content: for name in \ sree\ rama\ laila\ srihari\ vicky\ john do echo $name done I need to remove all the name lines that exist between for (first line) and do line so that i can replace with new names. Output file should look like: (3 Replies)
Discussion started by: raosr020
3 Replies

2. Shell Programming and Scripting

Remove lines from one file that exist in another file

Hello Everyone, I'm currently have a requirement where I've generated a list of files with specific attributes and I need to know what lines are similar between the two files. For example: -File 1- line1 line2 line3 -File 2- line1 line2 line4 line5 -Desires Output- line1 line2... (5 Replies)
Discussion started by: omnivir
5 Replies

3. Shell Programming and Scripting

deleting lines from file

We have a server that logs transactions to a file. I want to write a script that will delete the first 50 lines of the file daily without renameing the file or moving the file. (8 Replies)
Discussion started by: daveisme
8 Replies

4. Shell Programming and Scripting

Removing Lines if value exist in first file

I tried a few ways to resolve this using a bash script w/ a loop, no luck. File1: roughly 6,000 account numbers such as: 1111 1512 1113 123 I also have a dozen or so csv files, w/ the account number in the 4th field. What I would like to do is remove all lines if the... (19 Replies)
Discussion started by: svn
19 Replies

5. UNIX for Advanced & Expert Users

Deleting lines from a file

How I can delete 100 lines anywhere in a file without opening a file and without renaming the file. (11 Replies)
Discussion started by: Nirgude07
11 Replies

6. Shell Programming and Scripting

Deleting lines from file using another file

I am having a requirement like File A contains All the records eg., ONE TWO THREE FOUR FIVE SIX SEVEN EIGHT NINE TEN File B contains invalid records of File A eg., TWO FOUR FIVE SEVEN (7 Replies)
Discussion started by: dinesh1985
7 Replies

7. UNIX for Dummies Questions & Answers

Deleting whole lines from a file

I have a file with 65 sets of 35 coordinates, and would like to isolate these coordinates so that I can easily copy the coordinates to another file. The problem is, I've got a 9 line header before each set of coordinates (so each set is 44 lines long). There are a zillion threads out there about... (3 Replies)
Discussion started by: red baron
3 Replies

8. Shell Programming and Scripting

Deleting lines inside a file without opening the file

Hi, Just consider there are around 10 lines in a file. Now is it possible to delete the first 2 lines in the file without opening the file. No matter whatever the content of the file is, I just wanna delete the first 2 lines without opening the file. Is that possible? If so, please help me out.... (3 Replies)
Discussion started by: toms
3 Replies

9. Shell Programming and Scripting

Deleting lines in a file

How do I delete all the lines after the line containing text ***DISCLOSURES*** . I want to delete this line too. Thank you (2 Replies)
Discussion started by: reachsamir
2 Replies

10. Shell Programming and Scripting

Deleting last 2 lines from the file.

Hi I have a file & always I need to remove or delete last 2 lines from that file. So in a file if I have 10 lines then it should return me first 8 lines. Can someone help me? (4 Replies)
Discussion started by: videsh77
4 Replies
Login or Register to Ask a Question