Hi!
I have concatenated two files which are wordlists, i.e., one word per line. The new file contains some doubles, but I cannot use sort and uniq as I need to keep the sort order that it is already in, which is not alphabetical, and uniq only compares adjacent lines, and the doubles are not on adjacent lines. Is there another simple way to remove doubles without altering the sort order? Unfortunately, there is no common pattern I can use to pick them out.

awk '!arr[$0]++' wordlist_file

Hey bipinajith, thanks for your reply! Would you mind explaining how that pattern works? I thought I knew a little about regexes, but I've never seen anything like that.

see the explanation:
http://www.unix.com/302678079-post2.html

I'd be more worried if you had, as it's not a regex. It's more like C than anything.

It's an array with a string as the index. It checks if it's zero, then adds to it. The first time the index appears, it will print, the next times it won't.

Whew! I kinda think I get it. At least, until I try to type out my own explanation. You know, I think I'm going to read something about awk and come back tomorrow.

Look up associative array: Associative array - Wikipedia, the free encyclopedia