I am. What would not be fun is re-writing the merge section of GNU sort to make special use of an SSD, and verifying that it is correct and robust. I don't have a spare SSD anyway, so it wouldn't help. My current SSD is about full, though it does have a 32GB swap area.

I do have more test results. I finished another sort of my large file, using a batch-size parameter large enough to merge all of the temporaries at once. This meant increasing the default of 16 to at least 117. I chose 135. The default sort took 21.1 hours. The tweaked sort took 18.2, about a 14% improvement. I'll also check the effect of making smaller temporaries (reduce the in-RAM merge footprint) and widening the file merge some more. This will require figuring out the relationship of parameters to temporary size -- I already know it's not identical; it appears the buffer-size I request is the size of some internal structure, not the size of the temporary.

Of the 18.2 hours that sort took, 8.1 were user time (doing the compares, presumably,) and 2.2 were system time (making I/O calls, managing buffers, handling TLB misses, page faults and such). The remainder, another 8 hours or so, is about double the unaccounted time in a pure copy. I suspect the difference is the additional time in doing head seeks on disk that are not needed by the copy of un-fragmented files. I consider them I/O time.

Without an SSD big enough to hold all the temporaries, I don't see how to reduce that I/O time. The suggestions I've seen have not painted a coherent picture for me of how that would go with less I/O than I have now. The first stage of merge sorts are already in RAM and read and write sequentially on separate spindles. It's just the file merges that do the seeking, and that involves all of the data. How are you going to do that without seeking a drive large enough to hold it all? Nobody needs to answer that, because I don't even have the smaller SSD.

My data is not at all uniform; someone had surmised that it was in comtemplating a radix sort. Instead, it clusters like mad in ways that vary between datasets. So the sort buckets would be of unpredictable sizes and require space allocation, and I'm anyway afraid it could be hard to implement in two passes over the data, which I think is required if it's going to beat the GNU sort, and so could have much the same seeking behavior.

When you think about this, consider my largest drives are 2 TB, and my input data occupies more than half that space. And this is not the largest dataset I'm going to have (I'm just starting the main project). Any approach that writes a bunch of intermediate collections of data is going to be spread all over the disk, whether it's in one file or many. It's gonna seek.

---------- Post updated 11-25-14 at 09:49 AM ---------- Previous update was 11-24-14 at 08:21 PM ----------

And in a victory of data over speculation, I've pretty much convinced myself that attempting to create longer temporaries by sorting more stuff in RAM, let alone SSD, is counterproductive. The new results come from changing the parameters from simply

Code:

 export param='--batch-size=135'

to

Code:

export param='--batch-size=320 --buffer-size=11g'

This results in temporaries about half the size, and an improvement in overall time from 18.2 hours to 11.8, just from using shorter temporaries (and increasing the batch so they're all merged at once.)

The user time did not change that much, though it did decrease from 8.7 to 7.6 hours. System time stayed about the same. I don't know why the unaccounted real time decreased by about 6.5 hours. Since it contradicts my intuition, I'll run a few more tests with other parameters to verify the trend.

If the trend holds up, I'll be decreasing the temporary size as much as possible, and making corresponding increases in batch size, until I reach a limit or a sweet spot.