The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
Remove Duplicate Lines in File Remove Duplicate Lines in File
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Remove duplicate lines in log files karthikn7974 Shell Programming and Scripting 4 03-21-2009 06:41 PM
Remove all instances of duplicate records from the file vukkusila Shell Programming and Scripting 3 12-12-2007 07:50 AM
Remove Duplicate lines from File Nysif Steve UNIX for Dummies Questions & Answers 18 09-09-2007 08:57 AM
how to remove duplicate lines fredao Shell Programming and Scripting 3 12-13-2006 12:51 PM
Duplicate lines in the file guptan UNIX for Advanced & Expert Users 3 05-18-2006 05:28 AM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish
 
LinkBack Thread Tools Search this Thread Rating: Thread Rating: 1 votes, 5.00 average. Display Modes
  #1 (permalink)  
Old 07-25-2005
Teh Tiack Ein Teh Tiack Ein is offline
Registered User
  
 

Join Date: Jul 2005
Posts: 7
Remove Duplicate Lines in File
I am doing KSH script to remove duplicate lines in a file. Let say the file has format below.

FileA
------
1253-6856
3101-4011
1827-1356
1822-1157
1822-1157
1000-1410
1000-1410
1822-1231
1822-1231
3101-4011
1822-1157
1822-1231
........

and I want to simply it with no duplicate line as file below.

FileA
------
1253-6856
3101-4011
1827-1356
1822-1157
1000-1410
1822-1231
........

How can I do that in KSH?
  #2 (permalink)  
Old 07-25-2005
nir_s nir_s is offline
Registered User
  
 

Join Date: Jun 2004
Posts: 148
Try the following command:

Code:
awk '!x[$0]++' file > file.new
Nir
  #3 (permalink)  
Old 07-25-2005
Teh Tiack Ein Teh Tiack Ein is offline
Registered User
  
 

Join Date: Jul 2005
Posts: 7
Thanks Nir! This is awesome!
  #4 (permalink)  
Old 07-25-2005
pixelbeat pixelbeat is offline
Registered User
  
 

Join Date: Jun 2005
Location: Ireland
Posts: 61
uniq file > file.new

If the files are not already sorted then:

sort file | uniq > file.new
  #5 (permalink)  
Old 07-26-2005
ssk ssk is offline
Registered User
  
 

Join Date: May 2005
Posts: 25
Smile
sort -u filename > filename.new
  #6 (permalink)  
Old 01-12-2006
sumitgarg sumitgarg is offline
Registered User
  
 

Join Date: Feb 2005
Posts: 18
Thanks
Thanks ssk!

Your solution is so simple and so effective!


Thanks,

Sumit Garg
Sponsored Links
Closed Thread

Bookmarks

« cat behavior when passing http prams as arg | Searching a string for a character »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Linear Mode Linear Mode
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 09:32 AM.

The UNIX and Linux Forums - Learn UNIX and UNIX Commands RSS Feeds - Contact Us - The UNIX and Linux Forums - Learn UNIX and UNIX Commands - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language translation by Google.
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0