The UNIX and Linux Forums  

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
Removing dupicate lines in the file ..(they are not continuous) Removing dupicate lines in the file ..(they are not continuous)
Google UNIX.COM
Forums Portal Register Forum Rules FAQContribute Members List Arcade Search Today's Posts Mark Forums Read


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Removing the first and last lines in a file naveendronavall Shell Programming and Scripting 2 12-29-2007 10:22 PM
Removing the first and last lines in a file naveendronavall AIX 1 12-29-2007 08:44 PM
Removing lines from a file computersaysno UNIX for Dummies Questions & Answers 6 11-14-2006 03:50 PM
Removing lines within a file tookers Shell Programming and Scripting 3 08-22-2006 06:49 AM
Removing lines from a file PradeepRed Shell Programming and Scripting 4 12-17-2005 03:34 AM

Reply
Page 1 of 2 1 2 >
 
Submit Tools LinkBack Thread Tools Search this Thread Display Modes
  #1  
Old 09-29-2005
Registered User
  

Join Date: Aug 2005
Posts: 23
Removing dupicate lines in the file ..(they are not continuous)
I have duplicates records in a file, but they are not consecutive. I want to remove the duplicates , using script.

Can some one help me in writing a ksh script to implement this task.



Ex file is like below.



1234

5689

4556

1234

4444

5555

6889

5689

7898

1234





from the above file i want to get rid of


duplicates of 1234 and 5689



thanks

srini
Reply With Quote
Forum Sponsor
  #2  
Old 09-29-2005
...@...
  

Join Date: Feb 2004
Location: NM
Posts: 4,298
Code:
awk '!x[$0]++' file.old >file.new
or
Code:
sort -u -o file.new file.old
Reply With Quote
  #3  
Old 09-29-2005
vgersh99's Avatar
Moderator
  

Join Date: Feb 2005
Location: Boston, MA
Posts: 3,029
Code:
nawk '!a[$0]' file.txt
Reply With Quote
  #4  
Old 09-29-2005
Technorati Master
  

Join Date: Mar 2005
Location: Large scale systems...
Posts: 2,610
to eliminate all the duplicates
Code:
sort <filename> | uniq -u
to supress only the duplicates
Code:
sort <filename> | uniq
Last edited by matrixmadhan; 09-29-2005 at 07:04 AM. Reason: reread the problem
Reply With Quote
  #5  
Old 09-29-2005
Registered User
  

Join Date: Aug 2005
Posts: 23
It works fine , but
Thank you everyone, it works fine ,

But I have some other problem now , The result file is coming as

Sep 28 11:09:33> Error on trans: 2
Sep 28 12:10:42> Error on trans: 1
Sep 28 12:10:43> Error on trans: 1
Sep 28 12:14:43> Error on trans: 1
Sep 28 12:14:44> Error on trans: 1

I want the output to be as follows


Sep 28 11:09:33> Error on trans: 2
Sep 28 12:10:42> Error on trans: 1



Can some one help me in doing this too.

thanks
Srinii
Reply With Quote
  #6  
Old 09-29-2005
vgersh99's Avatar
Moderator
  

Join Date: Feb 2005
Location: Boston, MA
Posts: 3,029
Code:
nawk '!a[$NF]++' file.txt
Reply With Quote
  #7  
Old 09-29-2005
Registered User
  

Join Date: Aug 2005
Posts: 23
Thanks, It works ..
hi vgersh99,


Thanks for your help,

Can you please explan me what actually is happening when we execute the above command ??


i.e nawk '!a[$NF]++' file.txt

Thanks
Srini
Reply With Quote
Google The UNIX and Linux Forums
Reply
Page 1 of 2 1 2 >

« xargs doubt | Trouble using substr function with Bourne shell script »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Linear Mode Linear Mode




All times are GMT -7. The time now is 11:59 PM.

The UNIX and Linux Forums RSS Feeds - Contact Us - The UNIX and Linux Forums - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited.
The UNIX and Linux Forums Content Copyright ©1993-2008. All Rights Reserved.Ad Management by RedTyger Visit The Complex Event Processing Blog

Content Relevant URLs by vBSEO 3.2.0