UNIX for Dummies Questions & Answers

I was wondering how sort works.

Does file size and time to sort increase geometrically?

I have a 5.3 billion line file I'd like to use with sort -u I'm wondering if that'll take forever because of a geometric expansion?

If it takes 100 hours that's fine but not 100 days.

Thanks so much.

I've worked with very large files before but not quite that large. Given several million lines, I found I could reduce the overall time by splitting the file into smaller units using grep, sort and save each unit, then combine the results.

For example, given a file of cksum output which will always begin with a numeric check sum,

Code:

#!/bin/ksh

    I=0
    while [ $I -lt 10 ]
    do
        grep "^$I" file | sort -u > $I.sort
        (( I += 1 ))
    done

    cat ?.sort > file.sorted

This could, of course, be expanded to include any characters replacing the while loop with a for loop.

If the file is somewhat redundant and the key is not too huge try awk. Let's say the first two words of the record make the key:

Code:

awk '!arr[$1 $2]++' hugefile > uniquefile

This reads thru the file once, no sorting. Obviously there has to be enough disk space for the outputfile. It uses associative arrays - hash tables is another name.