Remove Duplicate Lines in File

Teh Tiack Ein · #1 07-25-2005

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below.

FileA
------
1253-6856
3101-4011
1827-1356
1822-1157
1822-1157
1000-1410
1000-1410
1822-1231
1822-1231
3101-4011
1822-1157
1822-1231
........

and I want to simply it with no duplicate line as file below.

FileA
------
1253-6856
3101-4011
1827-1356
1822-1157
1000-1410
1822-1231
........

How can I do that in KSH?

nir_s · #2 07-25-2005

Try the following command:

Code:

awk '!x[$0]++' file > file.new

Teh Tiack Ein · #3 07-25-2005

Thanks Nir! This is awesome!

pixelbeat · #4 07-25-2005

uniq file > file.new

If the files are not already sorted then:

sort file | uniq > file.new

ssk · #5 07-26-2005

sort -u filename > filename.new

sumitgarg · #6 01-12-2006

Thanks ssk!

Your solution is so simple and so effective!

Thanks,

Sumit Garg

Thread Tools	Search this Thread
Show Printable Version Email this Page Subscribe to this Thread	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Remove duplicate lines in log files	karthikn7974	Shell Programming and Scripting	4	03-21-2009 05:41 PM
Remove all instances of duplicate records from the file	vukkusila	Shell Programming and Scripting	3	12-12-2007 06:50 AM
Remove Duplicate lines from File	Nysif Steve	UNIX for Dummies Questions & Answers	18	09-09-2007 08:57 AM
how to remove duplicate lines	fredao	Shell Programming and Scripting	3	12-13-2006 11:51 AM
Duplicate lines in the file	guptan	UNIX for Advanced & Expert Users	3	05-18-2006 05:28 AM

The Following User Says Thank You to ssk For This Useful Post:
figaro (10-10-2010)