The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
how to delete duplicate rows in a file how to delete duplicate rows in a file
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
How to delete particular rows from a file suresh3566 Shell Programming and Scripting 5 06-02-2008 05:07 AM
duplicate rows in a file infyanurag Shell Programming and Scripting 3 05-22-2008 12:39 AM
Delete repeated rows from a file tonet Shell Programming and Scripting 8 04-08-2008 09:42 AM
delete semi-duplicate lines from file? paqman Shell Programming and Scripting 2 02-11-2008 05:46 PM
Delete Duplicate records from a tilde delimited file irshadm Shell Programming and Scripting 5 12-06-2007 05:36 AM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 06-18-2008
vamshikrishnab vamshikrishnab is offline
Registered User
  
 

Join Date: May 2008
Posts: 32
how to delete duplicate rows in a file
I have a file content like below.

Code:
"0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCYO","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCYO","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCYO","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCYO","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCZI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCZI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCZI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCZI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCZO","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCZO","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCZO","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCZO","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""

I want to know the command to delete duplicate rows in whole file.

Thanks in advance.
Last edited by Yogesh Sawant; 06-18-2008 at 09:13 AM.. Reason: added code tags
  #2 (permalink)  
Old 06-18-2008
DukeNuke2's Avatar
DukeNuke2 DukeNuke2 is offline Forum Staff  
Soulman
  
 

Join Date: Jul 2006
Location: Germany, Berlin
Posts: 2,953
have a look at the "sort" command...

sort -u /your/file > /new/file

can maybe help (untestet!)
  #3 (permalink)  
Old 06-18-2008
DukeNuke2's Avatar
DukeNuke2 DukeNuke2 is offline Forum Staff  
Soulman
  
 

Join Date: Jul 2006
Location: Germany, Berlin
Posts: 2,953
moved from "solaris" to "shell scripting" cause this is no solaris specific problem!
  #4 (permalink)  
Old 06-18-2008
bisla.yogender bisla.yogender is offline
Registered User
  
 

Join Date: Mar 2008
Posts: 30
man uniq
  #5 (permalink)  
Old 06-18-2008
joeyg's Avatar
joeyg joeyg is offline Forum Staff  
modérateur
  
 

Join Date: Dec 2007
Location: Home of 17-time world champion Boston Celtics
Posts: 1,311
Wink beware of order of data
If the file is already in sorted order, than one can simply do
Code:
uniq file1
However, if the file is in random order, you will not get a truly unique file output. I think it only looks to adjoining records, so you could have repeated entries in your output. Thus, probably better doing
Code:
sort -u file1
Both of the above commands send output to screen, thus to send to a file, try
Code:
sort -u file1 >file2
  #6 (permalink)  
Old 06-18-2008
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
  
 

Join Date: Feb 2004
Location: NM
Posts: 5,717
or if you like, an awk example
Code:
awk '!arr[$0]++' filename > newfile
Closed Thread

Bookmarks

Tags
solaris

« Selecting patterns from the results of grep | (probably stupidly simple) if [ $var1 = $var2 ] problem... »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Hybrid Mode Hybrid Mode
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 04:03 AM.

The UNIX and Linux Forums - Learn UNIX and UNIX Commands RSS Feeds - Contact Us - The UNIX and Linux Forums - Learn UNIX and UNIX Commands - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0