The UNIX and Linux Forums  

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
Remove Duplicate Lines in File Remove Duplicate Lines in File
Google UNIX.COM
Forums Portal Register Forum Rules FAQContribute Members List Arcade Search Today's Posts Mark Forums Read


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Remove duplicate lines in log files karthikn7974 Shell Programming and Scripting 3 1 Day Ago 04:17 PM
Remove all instances of duplicate records from the file vukkusila Shell Programming and Scripting 3 12-12-2007 04:50 AM
Remove Duplicate lines from File Nysif Steve UNIX for Dummies Questions & Answers 18 09-09-2007 05:57 AM
how to remove duplicate lines fredao Shell Programming and Scripting 3 12-13-2006 09:51 AM
Duplicate lines in the file guptan UNIX for Advanced & Expert Users 3 05-18-2006 02:28 AM

Reply
 
Submit Tools LinkBack Thread Tools Search this Thread Display Modes
  #1  
Old 07-24-2005
Registered User
  

Join Date: Jul 2005
Posts: 7
Remove Duplicate Lines in File
I am doing KSH script to remove duplicate lines in a file. Let say the file has format below.

FileA
------
1253-6856
3101-4011
1827-1356
1822-1157
1822-1157
1000-1410
1000-1410
1822-1231
1822-1231
3101-4011
1822-1157
1822-1231
........

and I want to simply it with no duplicate line as file below.

FileA
------
1253-6856
3101-4011
1827-1356
1822-1157
1000-1410
1822-1231
........

How can I do that in KSH?
Reply With Quote
Forum Sponsor
  #2  
Old 07-25-2005
Registered User
  

Join Date: Jun 2004
Posts: 146
Try the following command:

Code:
awk '!x[$0]++' file > file.new
Nir
Reply With Quote
  #3  
Old 07-25-2005
Registered User
  

Join Date: Jul 2005
Posts: 7
Thanks Nir! This is awesome!
Reply With Quote
  #4  
Old 07-25-2005
Registered User
  

Join Date: Jun 2005
Location: Ireland
Posts: 61
uniq file > file.new

If the files are not already sorted then:

sort file | uniq > file.new
Reply With Quote
  #5  
Old 07-26-2005
ssk ssk is offline
Registered User
  

Join Date: May 2005
Posts: 25
Smile
sort -u filename > filename.new
Reply With Quote
  #6  
Old 01-12-2006
Registered User
  

Join Date: Feb 2005
Posts: 18
Thanks
Thanks ssk!

Your solution is so simple and so effective!


Thanks,

Sumit Garg
Reply With Quote
Google The UNIX and Linux Forums
Reply

« cat behavior when passing http prams as arg | Searching a string for a character »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Linear Mode Linear Mode




All times are GMT -7. The time now is 05:14 AM.

The UNIX and Linux Forums RSS Feeds - Contact Us - The UNIX and Linux Forums - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited.
The UNIX and Linux Forums Content Copyright ©1993-2008. All Rights Reserved.Ad Management by RedTyger Visit The Complex Event Processing Blog

Content Relevant URLs by vBSEO 3.2.0