removing duplicate lines from a file

ocelot · #1 (**permalink**) 01-25-2007

Hi,

I am trying to remove duplicate lines from a file. For example the contents of example.txt is:

this is a test
2342
this is a test
34343
this is a test
43434

and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with:

2342
34343
43434

thanks

Krrishv · #2 (**permalink**) 01-25-2007

grep -v "^t" <filename>

anbu23 · #3 (**permalink**) 01-25-2007

Quote:

Originally Posted by ocelot

Hi,

I am trying to remove duplicate lines from a file. For example the contents of example.txt is:

this is a test
2342
this is a test
34343
this is a test
43434

and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with:

2342
34343
43434

thanks

Code:

grep -v "[[:alpha:]]" file

jim mcnamara · #4 (**permalink**) 01-25-2007

Code:

# this prints only one line for each distinct record in oldfile
awk '!x[$0]++' oldfile > newfile

# this completely removes all occurrances of all duplicated lines
awk 'x[$0]++ == 2' oldfile > t.sed
grep -v -f t.sed  oldfile > newfile

The other given solutions are data dependent, ie., using the special characteristics of duplicated data in your example to filter.

ocelot · #5 (**permalink**) 01-25-2007

thanks guys that worked a treat!

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Remove Duplicate lines from File	Nysif Steve	UNIX for Dummies Questions & Answers	18	09-09-2007 08:57 AM
How to redirect duplicate lines from a file????	zing_foru	UNIX for Dummies Questions & Answers	3	04-25-2007 07:03 AM
Duplicate lines in the file	guptan	UNIX for Advanced & Expert Users	3	05-18-2006 05:28 AM
Remove Duplicate Lines in File	Teh Tiack Ein	Shell Programming and Scripting	5	01-12-2006 08:30 AM
Removing duplicate lines ignore case	hellsd	UNIX for Dummies Questions & Answers	17	12-02-2004 10:47 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread: