I'm pretty sure that the buffer-size parameter controls only pass 1, which is where actual sorting occurs. It controls the size of the first set of temporary files. All subsequent passes are merges. Moreover, the default on my system is similar to what I get when I specify 4g, and is distinctly sub-optimal.

In testing, the gain was real but not big. The elephant in the room here seems to be the number of passes, and I/O time is a large percent of the total. While threading improves overlap, they're still competing for use of the same input and output files and directories.
Fair enough. But RAM is already 32GB, which maxes out my motherboard. Input, temporaries and output are already on three separate drives. Swap is separate and on an SSD, but I don't think I'm using it. So I'm squeezing out percentages. Fortunately, some of them are worth the effort; I'm already running about twice as fast as the built-in defaults, which have over-large buffers and not enough streams in a batch.