Remove Doubles Without Sort? | Unix Linux Forums | UNIX for Dummies Questions & Answers

  Go Back    


UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!

Remove Doubles Without Sort?

UNIX for Dummies Questions & Answers


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 12-11-2012
sudon't's Avatar
sudon't sudon't is offline
Registered User
 
Join Date: May 2012
Last Activity: 27 October 2014, 11:02 AM EDT
Location: The Cape Fear...ooooh!
Posts: 80
Thanks: 50
Thanked 0 Times in 0 Posts
Remove Doubles Without Sort?

Hi!
I have concatenated two files which are wordlists, i.e., one word per line. The new file contains some doubles, but I cannot use sort and uniq as I need to keep the sort order that it is already in, which is not alphabetical, and uniq only compares adjacent lines, and the doubles are not on adjacent lines. Is there another simple way to remove doubles without altering the sort order? Unfortunately, there is no common pattern I can use to pick them out.
Sponsored Links
    #2  
Old 12-11-2012
Yoda's Avatar
Yoda Yoda is offline Forum Advisor  
Jedi Master
 
Join Date: Jan 2012
Last Activity: 21 November 2014, 11:17 AM EST
Location: Galactic Empire
Posts: 3,389
Thanks: 235
Thanked 1,210 Times in 1,136 Posts

Code:
awk '!arr[$0]++' wordlist_file

The Following User Says Thank You to Yoda For This Useful Post:
sudon't (12-12-2012)
Sponsored Links
    #3  
Old 12-11-2012
sudon't's Avatar
sudon't sudon't is offline
Registered User
 
Join Date: May 2012
Last Activity: 27 October 2014, 11:02 AM EDT
Location: The Cape Fear...ooooh!
Posts: 80
Thanks: 50
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by bipinajith View Post
Code:
awk '!arr[$0]++' wordlist_file

Hey bipinajith, thanks for your reply! Would you mind explaining how that pattern works? I thought I knew a little about regexes, but I've never seen anything like that.
    #4  
Old 12-11-2012
rdcwayx rdcwayx is offline Forum Advisor  
Use nawk in Solaris
 
Join Date: Jun 2006
Last Activity: 22 March 2014, 12:27 PM EDT
Posts: 2,759
Thanks: 44
Thanked 418 Times in 406 Posts
see the explanation:
http://www.unix.com/302678079-post2.html
The Following User Says Thank You to rdcwayx For This Useful Post:
sudon't (12-12-2012)
Sponsored Links
    #5  
Old 12-11-2012
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 21 November 2014, 5:42 PM EST
Location: Saskatchewan
Posts: 19,902
Thanks: 845
Thanked 3,413 Times in 3,198 Posts
Quote:
Originally Posted by sudon't View Post
I thought I knew a little about regexes, but I've never seen anything like that.
I'd be more worried if you had, as it's not a regex. It's more like C than anything.

It's an array with a string as the index. It checks if it's zero, then adds to it. The first time the index appears, it will print, the next times it won't.
The Following User Says Thank You to Corona688 For This Useful Post:
sudon't (12-12-2012)
Sponsored Links
    #6  
Old 12-11-2012
sudon't's Avatar
sudon't sudon't is offline
Registered User
 
Join Date: May 2012
Last Activity: 27 October 2014, 11:02 AM EDT
Location: The Cape Fear...ooooh!
Posts: 80
Thanks: 50
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by rdcwayx View Post
Whew! I kinda think I get it. At least, until I try to type out my own explanation. You know, I think I'm going to read something about awk and come back tomorrow.
Sponsored Links
    #7  
Old 12-12-2012
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 21 November 2014, 4:36 PM EST
Location: NM
Posts: 10,279
Thanks: 287
Thanked 814 Times in 756 Posts
Look up associative array: Associative array - Wikipedia, the free encyclopedia
The Following User Says Thank You to jim mcnamara For This Useful Post:
sudon't (12-12-2012)
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
awk syntax mistake doubles desired output awknewb123 Shell Programming and Scripting 6 03-07-2012 01:49 PM
remove duplicates and sort dvah Shell Programming and Scripting 6 04-06-2011 12:14 PM
need help writing a program to look for doubles rickym2626 UNIX Desktop for Dummies Questions & Answers 2 04-13-2009 04:46 PM
How to remove duplicate records with out sort svenkatareddy Shell Programming and Scripting 19 06-11-2008 03:10 PM
long doubles crashnburn Programming 1 12-19-2002 11:12 PM



All times are GMT -4. The time now is 04:46 AM.