Search Results

Does your data really have all those blank lines in it?

---------- Post updated at 02:27 PM ---------- Previous update was at 02:24 PM ----------

Assuming it doesn't actually have all those...

15 gigs is still too large to do a sort on.

Here's a smaller-scale version that gets column 3. I set the split size to 100K so I could do sensible testing on my 5 meg test data.

#!/bin/sh
...

Take a look at:
https://www.unix.com/unix-dummies-questions-answers/162370-help-deleting-specific-rows-text-file.html#post302534704
where I find duplicates and then do something with the duplicates...

Most of the ways to do this don't work on a 120 GIGABYTE datafile. :eek: Finding duplicates in particular, means checking the row against all other rows. The easy/fast/efficient ways depend on...