UNIX for Advanced & Expert Users

Not really.

If your files can fit in memory, either way you do it -- cache or sort buffers -- it will be held in RAM. Yes, too many files at once is bad since the disk has to seek them individually.

I don't think this has much to do with the size of the temp files as much as their number. Doing a lot of seeking on a non-SSD disk makes its performance really bad. I once measured a disk's random read performance with caching disabled -- hopping from one sector to another, reading then moving, gets you fifty kilobytes per second on a disk good for 100MB/s. Using all available RAM is about as bad as turning off caching, by the way. Worse actually -- memory pressure will start pushing out useful things, making programs wait pointlessly for bits of code to return to them at need.

Cache could also explain the differing times your runs take. If there's significant parts of your file left in cache, that could speed up the next run.

I'm testing with my idea of a small file: 13 GB. My targets are more like TB-sized. These are the cases that motivate optimizing GNU sort. For modest-sized files, I wouldn't bother.

Size and number are linked. If they're bigger, there are fewer of them. So would love for the cache to contain all of the data from files being merged. I don't know how to do it, but I'm hoping my data will guide me to it indirectly.

True. I do not do anything to clear the caches before each run, but the tests are done by scripts that always do them in the same order, so I would expect more uniformity than I've seen so far.

It doesn't work that way. It doesn't run 16384 individual merges simultaneously, then 4096 merges simultaneously, then 1024, etc etc. It always does the number you tell it to, as many times as it takes to process the list of things to do.

I don't think small sorts hurt you, especially since they're small enough to be cached. What hurts you are merges on too many files at once for the disk to seek between, reducing it's I/O throughput.

Remember you are trying to find a "sweet spot" where CPU use and disk throughput are both at peak -- where the system can sustain full disk and cpu use.

Check if you're eating into swap sometimes. Hitting swap could have severe performance penalties that throw off your tests.

You could also check out external sorting. Since you know the data is much larger than RAM, this may be higher performance since it's purely sequential. You can control exactly how much your disk is being hit by what when.

Basically, split your n-terabyte file into chunks small enough to fit purely in memory -- 50% of memory size is a good guess -- and sort them, then merge them. This will give you full control of how many files are being read when, to the point the limit becomes disk speed (as it always was, ideally).

That makes no sense to me. If I tell it to make 1GB temporaries, my 13GB test file will make 13 of them and probably merge just once. If I tell it to make 1MB temporaries, it will make 13,000 of them, and likely have to merge the outputs of the first set of merges at least one extra time, handling the same data more times with more disk I/O.

My real TB-sized inputs will see this even more.

BTW, on skimming the sort source code, I found that by default it tries to make the buffer-size parameter 1/8 of physical memory: 4GB. That was so slow that I started this investigation. Smaller turned out to be better, possibly because none of the TLB or caches could deal with it effectively during the initial in-RAM merges because of the huge working set.

Not sure what you mean about small sorts; I can't have them. I have large files, requiring large sorts. The files are large enough that I do not expect anything from previous merges to be available if the data has to be merged again, and I expect the data from different inputs to be far enough apart on disk to require seeks no matter what I do. Accordingly, I'd like to minimize the number of merge passes.

Even with these large sizes, sweet spots are exactly what I'm looking for.

I'm looking. But with the modest parameters I'm giving it, I hardly expect my 32-GB RAM to need to swap. The largest buffer-size I've ever tried is 4g, and that was already a bad idea and I don't do that any more. I'm distinctly sub-G now.

This is because we are talking about different problems. You are talking about how many files it must split into -- I am talking about how many files are being merged at once.

Merging fewer files at once will mean less disk thrashing but more passes. Since you do not have an SSD input, the performance penalties for sustained disk thrashing -- skipping from file1 to file2 to file3 to file4 back to file1 -- are very, very high, hundredsfold. This could also be throwing off your results if the disk only thrashes sometimes due to unfortunate coincidences of access and cache.

Using the SSD as temp space may still be useful too, even if it's smaller than your file. It's another independent "spindle" you could be using to merge partial files in, particularly with the 'external sort' idea. Create n 20-gig sorted files on your SSD, merge them into one big sorted file on your main spindle, rinse, repeat.

If you look at the first 10 characters in each file and create a file that represents each combination of characters for the first 10 characters, you would have 59,049 files, if my math is correct. You can then put each row from each file in the corresponding file. This when you sort each file, the file is already sorted to the 10th character and the remaining sort would be easier to do and easier to do in parallel. Since you would not want multiple processes appending lines to the same file at the same time the initial bucket sort should probably get done as a single process. Then each of the 59,049 files can get sorted with as many processes as you want. Plus each file will be smaller than what you have now.

You can also break up the files by using the first three characters as directory names, that way each directory three levels down would have 1/9th of the total files. This would also make it easier to bucket sort by more than 10 characters. I included a file of each possible combination of characters for the first 10 characters. This approach assumes an even distribution of data across the first 10 characters.

I woulds still load the data into a database though...