Shell Programming and Scripting

EXAMPLE: Processing a big file using more cores
       To process a big file or some output you can use --pipe to split up the
       data into blocks and pipe the blocks into the processing program.

       If the program is gzip -9 you can do:

       cat bigfile | parallel --pipe --recend '' -k gzip -9 >bigfile.gz

       This will split bigfile into blocks of 1 MB and pass that to gzip -9 in
       parallel. One gzip will be run per CPU core. The output of gzip -9 will
       be kept in order and saved to bigfile.gz

       gzip works fine if the output is appended, but some processing does not
       work like that - for example sorting. For this GNU parallel can put the
       output of each command into a file. This will sort a big file in
       parallel:

       cat bigfile | parallel --pipe --files sort | parallel -Xj1 sort -m {}
       ';' rm {} >bigfile.sort

       Here bigfile is split into blocks of around 1MB, each block ending in
       '\n' (which is the default for --recend). Each block is passed to sort
       and the output from sort is saved into files. These files are passed to
       the second parallel that runs sort -m on the files before it removes
       the files. The output is saved to bigfile.sort.