Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
Search Forums:



Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 07-25-2005
Registered User
 

Join Date: Jul 2005
Posts: 7
Thanks: 0
Thanked 0 Times in 0 Posts
Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below.

FileA
------
1253-6856
3101-4011
1827-1356
1822-1157
1822-1157
1000-1410
1000-1410
1822-1231
1822-1231
3101-4011
1822-1157
1822-1231
........

and I want to simply it with no duplicate line as file below.

FileA
------
1253-6856
3101-4011
1827-1356
1822-1157
1000-1410
1822-1231
........

How can I do that in KSH?
Sponsored Links
    #2  
Old 07-25-2005
Registered User
 

Join Date: Jun 2004
Posts: 165
Thanks: 0
Thanked 0 Times in 0 Posts
Try the following command:


Code:
awk '!x[$0]++' file > file.new

Nir
Sponsored Links
    #3  
Old 07-25-2005
Registered User
 

Join Date: Jul 2005
Posts: 7
Thanks: 0
Thanked 0 Times in 0 Posts
Thanks Nir! This is awesome!
    #4  
Old 07-25-2005
Registered User
 

Join Date: Jun 2005
Location: Ireland
Posts: 61
Thanks: 0
Thanked 0 Times in 0 Posts
uniq file > file.new

If the files are not already sorted then:

sort file | uniq > file.new
Sponsored Links
    #5  
Old 07-26-2005
ssk ssk is offline
Registered User
 

Join Date: May 2005
Posts: 25
Thanks: 0
Thanked 1 Time in 1 Post
Smile

sort -u filename > filename.new
The Following User Says Thank You to ssk For This Useful Post:
figaro (10-10-2010)
Sponsored Links
    #6  
Old 01-12-2006
Registered User
 

Join Date: Feb 2005
Posts: 18
Thanks: 0
Thanked 0 Times in 0 Posts
Thanks

Thanks ssk!

Your solution is so simple and so effective!


Thanks,

Sumit Garg
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Remove duplicate lines in log files karthikn7974 Shell Programming and Scripting 4 03-21-2009 05:41 PM
Remove all instances of duplicate records from the file vukkusila Shell Programming and Scripting 3 12-12-2007 06:50 AM
Remove Duplicate lines from File Nysif Steve UNIX for Dummies Questions & Answers 18 09-09-2007 08:57 AM
how to remove duplicate lines fredao Shell Programming and Scripting 3 12-13-2006 11:51 AM
Duplicate lines in the file guptan UNIX for Advanced & Expert Users 3 05-18-2006 05:28 AM



All times are GMT -4. The time now is 04:59 AM.