Unix/Linux Go Back    

Speed up extraction od tar.bz2 files using bash

Shell Programming and Scripting

Kindly Note - This is a Single User Post by Forum Member Corona688 Regarding:
Speed up extraction od tar.bz2 files using bash.
Please Follow The Primary Link Above to View the Full Discussion.

Old Unix and Linux 06-13-2017   -   Original Discussion by cmccabe
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
Join Date: Aug 2005
Last Activity: 17 November 2017, 9:25 PM EST
Location: Saskatchewan
Posts: 22,512
Thanks: 1,151
Thanked 4,273 Times in 3,946 Posts
The command drl suggested was not pbzip but in fact parallel. Instead of extracting multiple partial files from one tar, you can get several tars extracting at once.

Which is what my code is for, actually.

I neglected one line at the end. It shouldn't have mattered, but if the code did manage to quit while the children were running, its possible it killed them instead of waiting. So:


maxproc=2 # Max number of threads.  Suggest 2, or 3 at most

# Count files
set -- /home/cmccabe/Desktop/NGS/API/*.tar.bz2

# Blank $1 $2 ...
set --

let i=1
for FILE in /home/cmccabe/Desktop/NGS/API/*.tar.bz2
        printf "(%2d/%2d)\tProcessing %s\n" "$i" "$FILES" "$FILE"
        let i=i+1

        tar -xvjf "$FILE" -C /home/cmccabe/Desktop/NGS/API >/dev/null &

        # Turn $1=pida $2=pidb $3=pidc $4=pidd, into
        #      $1=pida $2=pidb $3=pidc $4=pidd $5=pide
        set -- "$@" $!

        # Shift removes $1 and moves the rest down, so you get
        # $1=pidb $2=pidc $3=pidd $3=pide
        # $# is the number of arguments.
        if [ "$#" -ge $maxproc ]  ; then wait "$1" ; shift; fi


The Following User Says Thank You to Corona688 For This Useful Post:
cmccabe (06-15-2017)