Search Results

Using mmap64() means you only need components of a hash map of offsets on the heap. If the hash is new, it is unique so far, can be written and the offset needs to be remembered under that hash. If...

Any DBAs around with some spare disk space?

Use a DB server. Create a single column table with a unique index on that column. Insert each line as a row into the table, ignoring duplicate entry...

The sort -u has less scaling difficulty than associative arrays in a shell. If the order of lines is important, number them first, then sort out the duplicates, then sort it back to original order...

The problems are:

Using associative arrays in awk can use lots of memory for a file that large, so it may not work well. If the key(s) are a small size you might get it to work. And awk is...