Very interesting.

I had completely missed the --parallel option in reading the man page. I use this machine for everything, so I'm not sure I'll always want to swamp the cores, but I'll surely use some. I think this will help just the first pass -- when the temp files are first created -- because I'm going to try really hard to tweak parameters to make the second pass the last one where there's a single output file.

I'm just starting, and my largest input file is 1.3 TB. Therefore, SSD drives are too expensive for me; the only way to use just the one I already own and is not cluttered with my stuff already would be to chop the sort into many segments -- and then the merge phase would have to be all on spinning drives anyway. Maybe in 10 years.

I may compress the records eventually, but I'm running tests now that show that user time is at most 25% of the total time. That limits the effectiveness. Plus, each run is a one-time thing, and I worry about the overhead of the compression since it has to be a separate process.

I've got a script that is testing combinations of batch-size, buffer-size, compress-program (with various levels of compressions) and I'll add --parallel to it. I've been curious about sort for some years, at this seems to be the moment when I really want to investigate.

---------- Post updated at 02:07 PM ---------- Previous update was at 11:54 AM ----------

An excellent idea for a cache-aware in-memory sort. I have never before seen this idea extended to VM like this. However, it won't work for my TB-scale files without TB's of swap space. I'm also not quite ready to rewrite GNU sort to use the idea.

Thinking of alternatives, I read somewhere that phase 1 of the existing sort used qsort on a full buffer, and spits it out as the result if it's small and as temporaries otherwise. I know qsort is a favorite because it's unusually good in it's average performance, and is in-place. I once wrote a sort that used a heapsort algorithm in phase 1 to produce longer temporaries -- averaging twice the size of the working storage, with guaranteed n-log-n performance. Too bad it wouldn't be a good general-purpose replacement -- it incurs memory fragmentation when using variable-length records. The qsort implementation doesn't because it cleans out the workspace on every buffer, and the heapsort method can't do that and still generate the long outputs.

---------- Post updated at 02:13 PM ---------- Previous update was at 02:07 PM ----------

I'm now running some tests with a "short" 14 GB file. Some extra eyes on the script would be appreciated, as I've already been caught in one error already in this thread.

Code:

#!/bin/bash
# find times for sorting as compared to copying time.

ulimit -Sn hard
export TMPDIR=/mnt/tmptmp
export DATA=s2740.posn
export DEST=/mnt/h2/tmp/junk
export TEMP=/mnt/tmptmp/junk
export LC_ALL=C

cd /mnt/h1/posn
echo prep
df -h .
df -h /mnt/tmptmp
df -h /mnt/h2/tmp

# do something to make the cache state consistent
rm -f $TEMP $DEST
bash -c 'cp $DATA $TEMP; cp $TEMP $DEST'
echo

echo -n first copy then second copy
rm -f $TEMP $DEST
time bash -c 'cp $DATA $TEMP'
time bash -c 'cp $TEMP $DEST'
echo

echo -n copy both
rm -f $TEMP $DEST
time bash -c 'cp $DATA $TEMP; cp $TEMP $DEST'
echo

echo -n sort with default params
rm -f $TEMP $DEST
time sort $DATA --output=$DEST
echo

echo -n sort with n=16, buffer=4g
rm -f $TEMP $DEST
time sort --buffer-size=4g --batch-size=16 $DATA --output=$DEST
echo


buf=1
while [ $buf -lt 16 ]
do
  bat=10
  while [ $bat -lt 101 ]
  do
    for par in 1 2 4
    do
      echo -n sort with parallel=$par buffer=${buf}g batch=$bat and gzip from none to 9
      rm -f $TEMP $DEST
      time sort --parallel=$par --buffer-size=${buf}g --batch-size=$bat $DATA --output=$DEST
      export GZIP='-1'
      rm -f $TEMP $DEST
      time sort --parallel=$par --buffer-size=${buf}g --batch-size=$bat $DATA --output=$DEST
      export GZIP='-2'
      rm -f $TEMP $DEST
      time sort --parallel=$par --buffer-size=${buf}g --batch-size=$bat $DATA --output=$DEST
      export GZIP='-3'
      rm -f $TEMP $DEST
      time sort --parallel=$par --buffer-size=${buf}g --batch-size=$bat $DATA --output=$DEST
      export GZIP='-4'
      rm -f $TEMP $DEST
      time sort --parallel=$par --buffer-size=${buf}g --batch-size=$bat $DATA --output=$DEST
      export GZIP='-5'
      rm -f $TEMP $DEST
      time sort --parallel=$par --buffer-size=${buf}g --batch-size=$bat $DATA --output=$DEST
      export GZIP='-6'
      rm -f $TEMP $DEST
      time sort --parallel=$par --buffer-size=${buf}g --batch-size=$bat $DATA --output=$DEST
      export GZIP='-7'
      rm -f $TEMP $DEST
      time sort --parallel=$par --buffer-size=${buf}g --batch-size=$bat $DATA --output=$DEST
      export GZIP='-8'
      rm -f $TEMP $DEST
      time sort --parallel=$par --buffer-size=${buf}g --batch-size=$bat $DATA --output=$DEST
      export GZIP='-9'
      rm -f $TEMP $DEST
      time sort --parallel=$par --buffer-size=${buf}g --batch-size=$bat $DATA --output=$DEST
      bat=$(( bat + 10 ))
      echo
    done
  done
  buf=$(( buf + 10 ))
done

---------- Post updated at 03:02 PM ---------- Previous update was at 02:13 PM ----------

Oops. I can already see I omitted the --compress-program parameter from the ones I wanted to use gzip. Consider it fixed.