Speed up extraction od tar.bz2 files using bash Post: 302999131

Sponsored Content

Top Forums Shell Programming and Scripting Speed up extraction od tar.bz2 files using bash Post 302999131 by Corona688 on Tuesday 13th of June 2017 06:50:08 PM

06-13-2017

Registered User

The command drl suggested was not pbzip but in fact parallel. Instead of extracting multiple partial files from one tar, you can get several tars extracting at once.

Which is what my code is for, actually.

I neglected one line at the end. It shouldn't have mattered, but if the code did manage to quit while the children were running, its possible it killed them instead of waiting. So:

Code:

#!/bin/bash

maxproc=2 # Max number of threads.  Suggest 2, or 3 at most
i=0

# Count files
set -- /home/cmccabe/Desktop/NGS/API/*.tar.bz2
FILES="$#"

# Blank $1 $2 ...
set --

let i=1
for FILE in /home/cmccabe/Desktop/NGS/API/*.tar.bz2
do
        printf "(%2d/%2d)\tProcessing %s\n" "$i" "$FILES" "$FILE"
        let i=i+1

        tar -xvjf "$FILE" -C /home/cmccabe/Desktop/NGS/API >/dev/null &

        # Turn $1=pida $2=pidb $3=pidc $4=pidd, into
        #      $1=pida $2=pidb $3=pidc $4=pidd $5=pide
        set -- "$@" $!

        # Shift removes $1 and moves the rest down, so you get
        # $1=pidb $2=pidc $3=pidd $3=pide
        # $# is the number of arguments.
        if [ "$#" -ge $maxproc ]  ; then wait "$1" ; shift; fi
done

wait

This User Gave Thanks to Corona688 For This Post:

Corona688

View Public Profile for Corona688

Visit Corona688's homepage!

Find all posts by Corona688

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

extracting from tar.bz2

hi could any body tell me how to extract .tar.bz2 files i tried using tar but in vain. i found bzip2 in googling but i could not find it on machine unix tru64 please suggest.

2. Shell Programming and Scripting

compare two tar.bz2

Hello, I am using a bash script to archive directories of text files located in ${root}: tar cf ${root}.tar ${root}* bzip2 ${root}.tar I'd like to compare the newly produced archive two.tar.bz2 with the second latest one.tar.bz2. cmp one.tar.bz2 two.tar.bz2 returns one.tar.bz2 two.tar.bz2...

3. Shell Programming and Scripting

Optimize/speed-up perl extraction

Hi, Is there a way I can extract my data faster. You know my data is 1.2 GB text file with 8Million rows with 38 columns/fields. Imagine how huge this is. How I can optimized the data extraction using perl. That is why I'm creating a script to filter only those informations that I need. Is...

4. Shell Programming and Scripting

How to unpack and install .tar.bz2 library ?

Hi, I am trying to unpack and install .tar.bz2 library. I was told to cd / and than tar -jxvf /source-of-library-file?...tar.bz2 to get files unpacked and installed into / Darius $ pwd / $ $ tar -jxvf /tmp/local/root/ncurses-dev-addon.tar.bz2 ncurses-dev-addon/...

5. Shell Programming and Scripting

Put one tar.bz2 file to another tar.bz2

Hi experts, I have two tar.bz2 file,: a.tar.bz2 and b.tar.bz2 I want to put a.tar.bz2 in to b.tar.bz2 eg: b.tar.bz2 only have one file named "b.c" contained I want it contain "b.c and a.tar.bz2" I don't want to decompress the b.tar.bz2 to achieve this, I try with "cat a.tar.bz2 >>...

6. UNIX and Linux Applications

Download firefox-19.0.2.tar.bz2

Does anyone know a reliable source to download firefox-19.0.2.tar.bz2 from? I would think you be able to download from firefox or mozilla somewhere. I haven't gotten anything useful from my google searches.

7. Ubuntu

Error messages while extracting tar.bz2 in Ubuntu

while extracting a tar.bz2 file using the command tar xjf git.tar.bz2 I received error messages that shows Cannot hard link to and Cannot create symlink to error messages what will be the reason for those error messages.

8. Programming

Aria2c to download and extract. tar.bz2

I am using aria2c to download a .tar.bz2 and trying to extract it in the same command. I can download the file but not extract it. I can also manually extract the tar.bz2., but not in the same command. Thank you :). aria2c -x8 -l log.txt -c -d /xx/xx/xxx --use-head=true --http-user "<user>" ...

9. Shell Programming and Scripting

Bash not removing all .tar.bz2 files after extracting

In the bash below each .tar.bz2 (usually 2) are extracted and then the original .tar.bz2 is removed. However, only one (presumably the first extracted) is being removed, however both are extracted. I am not sure why this is? Thank you :). tar.bz2 folders in /home/cmccabe/Desktop/NGS/API ...

LEARN ABOUT DEBIAN

pristine-bz2

PRISTINE-BZ2(1) 						   pristine-bz2 						   PRISTINE-BZ2(1)

NAME

       pristine-bz2 - regenerate pristine bz2 files

SYNOPSIS

       pristine-bz2 [-vdk] gendelta file.bz2 delta

       pristine-bz2 [-vdk] genbz2 delta file

DESCRIPTION

       This is a complement to the pristine-tar(1) command. Normally you don't need to run it by hand, since pristine-tar calls it as necessary to
       handle .tar.bz2 files.

       pristine-bz2 gendelta takes the specified bz2 file, and generates a small binary delta file that can later be used by pristine-bz2 genbz2
       to recreate the original file.

       pristine-bz2 genbz2 takes the specified delta file, and compresses the specified input file (which must be identical to the contents of the
       original bz2 file). The resulting file will be identical to the original gz file used to create the delta.

       The approach used to regenerate the original bz2 file is to figure out how it was produced -- what compression level was used, whether it
       was built with bzip2(1) or with pbzip2(1).

       Note that other tools exist, like bzip2smp or dbzip2, but they are said to be bit-identical with bzip2. Anyway, bzip2 looks like the most
       widespread implementation, so it's hard to find bzip2 files that make pristine-bz2 fail. Please report!

       The deprecated bzip1 compression method hasn't been implemented.

       If the delta filename is "-", pristine-bz2 reads or writes it to stdio.

OPTIONS

       -v  Verbose mode, show each command that is run.

       -d  Debug mode.

       -k  Don't clean up the temporary directory on exit.

       -t  Try harder to determine how to generate deltas of difficult bz2 files.

ENVIRONMENT

       TMPDIR
	   Specifies a location to place temporary files, other than the default.

AUTHOR

       Joey Hess <joeyh@debian.org>, Faidon Liambotis <paravoid@debian.org>, Cyril Brulebois <cyril.brulebois@enst-bretagne.fr>

       Licensed under the GPL, version 2.

perl v5.14.2							    2013-06-01							   PRISTINE-BZ2(1)