UNIX for Advanced & Expert Users

What about a regular sort and then awk:

Code:

sort infile | awk '$1!=p; {p=$1}'

--
Never mind, It doesn't make much of a difference...

Using mmap64() means you only need components of a hash map of offsets on the heap. If the hash is new, it is unique so far, can be written and the offset needs to be remembered under that hash. If not new, the actual keys are compared between the two mmap'd locations. If it is a miss, it is added as a second record in the hash bucket. The empty hash map is an array on N null hash bucket pointers, where keys are hashed modulo N. If you write your own container, you can make N modulo 2 so the modulus operation is a mask of low bits. On 50 GB file with 50 byte records and a 4 GHZ CPU, every 4 cycles per record is a second of run time.

Conversely, some sorts can use parallel resources. Professional tools like abInitio divide up files for parallel processing. While record boundaries are not predictable, the threads know that they are responsible for only new lines above their starting point and possibly past their end point. Eventually, merging the sorts generally bottlenecks things, unless input threads send records to N key-segmented output sort threads. Then, each output is just concatenated. SyncSort makes an app out of it.

Probably an iteration?
Assuming awk can handle 10000 lines.

Code:

awk '{if (NR<10000) {if (! s[$0]++) print >> uniq} else {if (! $0 in s) print}}' < file1 > file2

Then repeat with

Code:

awk '...' < file2 > file1

...

The result is in the file uniq.