Unix/Linux Go Back    



Speed up extraction od tar.bz2 files using bash

Shell Programming and Scripting




Kindly Note - This is a Single User Post by Forum Member Corona688 Regarding:
Speed up extraction od tar.bz2 files using bash.
Please Follow The Primary Link Above to View the Full Discussion.

   
Old Unix and Linux 06-13-2017
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 19 September 2017, 5:41 PM EDT
Location: Saskatchewan
Posts: 22,391
Thanks: 1,126
Thanked 4,230 Times in 3,911 Posts
The command drl suggested was not pbzip but in fact parallel. Instead of extracting multiple partial files from one tar, you can get several tars extracting at once.

Which is what my code is for, actually.

I neglected one line at the end. It shouldn't have mattered, but if the code did manage to quit while the children were running, its possible it killed them instead of waiting. So:


Code:
#!/bin/bash

maxproc=2 # Max number of threads.  Suggest 2, or 3 at most
i=0

# Count files
set -- /home/cmccabe/Desktop/NGS/API/*.tar.bz2
FILES="$#"

# Blank $1 $2 ...
set --

let i=1
for FILE in /home/cmccabe/Desktop/NGS/API/*.tar.bz2
do
        printf "(%2d/%2d)\tProcessing %s\n" "$i" "$FILES" "$FILE"
        let i=i+1

        tar -xvjf "$FILE" -C /home/cmccabe/Desktop/NGS/API >/dev/null &

        # Turn $1=pida $2=pidb $3=pidc $4=pidd, into
        #      $1=pida $2=pidb $3=pidc $4=pidd $5=pide
        set -- "$@" $!

        # Shift removes $1 and moves the rest down, so you get
        # $1=pidb $2=pidc $3=pidd $3=pide
        # $# is the number of arguments.
        if [ "$#" -ge $maxproc ]  ; then wait "$1" ; shift; fi
done

wait

The Following User Says Thank You to Corona688 For This Useful Post:
cmccabe (06-15-2017)