Speed up extraction od tar.bz2 files using bash Post: 302999131

Sponsored Content

Top Forums Shell Programming and Scripting Speed up extraction od tar.bz2 files using bash Post 302999131 by Corona688 on Tuesday 13th of June 2017 06:50:08 PM

06-13-2017

Registered User

The command drl suggested was not pbzip but in fact parallel. Instead of extracting multiple partial files from one tar, you can get several tars extracting at once.

Which is what my code is for, actually.

I neglected one line at the end. It shouldn't have mattered, but if the code did manage to quit while the children were running, its possible it killed them instead of waiting. So:

Code:

#!/bin/bash

maxproc=2 # Max number of threads.  Suggest 2, or 3 at most
i=0

# Count files
set -- /home/cmccabe/Desktop/NGS/API/*.tar.bz2
FILES="$#"

# Blank $1 $2 ...
set --

let i=1
for FILE in /home/cmccabe/Desktop/NGS/API/*.tar.bz2
do
        printf "(%2d/%2d)\tProcessing %s\n" "$i" "$FILES" "$FILE"
        let i=i+1

        tar -xvjf "$FILE" -C /home/cmccabe/Desktop/NGS/API >/dev/null &

        # Turn $1=pida $2=pidb $3=pidc $4=pidd, into
        #      $1=pida $2=pidb $3=pidc $4=pidd $5=pide
        set -- "$@" $!

        # Shift removes $1 and moves the rest down, so you get
        # $1=pidb $2=pidc $3=pidd $3=pide
        # $# is the number of arguments.
        if [ "$#" -ge $maxproc ]  ; then wait "$1" ; shift; fi
done

wait

This User Gave Thanks to Corona688 For This Post:

Corona688

View Public Profile for Corona688

Visit Corona688's homepage!

Find all posts by Corona688

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

extracting from tar.bz2

hi could any body tell me how to extract .tar.bz2 files i tried using tar but in vain. i found bzip2 in googling but i could not find it on machine unix tru64 please suggest.

2. Shell Programming and Scripting

compare two tar.bz2

Hello, I am using a bash script to archive directories of text files located in ${root}: tar cf ${root}.tar ${root}* bzip2 ${root}.tar I'd like to compare the newly produced archive two.tar.bz2 with the second latest one.tar.bz2. cmp one.tar.bz2 two.tar.bz2 returns one.tar.bz2 two.tar.bz2...

3. Shell Programming and Scripting

Optimize/speed-up perl extraction

Hi, Is there a way I can extract my data faster. You know my data is 1.2 GB text file with 8Million rows with 38 columns/fields. Imagine how huge this is. How I can optimized the data extraction using perl. That is why I'm creating a script to filter only those informations that I need. Is...

4. Shell Programming and Scripting

How to unpack and install .tar.bz2 library ?

Hi, I am trying to unpack and install .tar.bz2 library. I was told to cd / and than tar -jxvf /source-of-library-file?...tar.bz2 to get files unpacked and installed into / Darius $ pwd / $ $ tar -jxvf /tmp/local/root/ncurses-dev-addon.tar.bz2 ncurses-dev-addon/...

5. Shell Programming and Scripting

Put one tar.bz2 file to another tar.bz2

Hi experts, I have two tar.bz2 file,: a.tar.bz2 and b.tar.bz2 I want to put a.tar.bz2 in to b.tar.bz2 eg: b.tar.bz2 only have one file named "b.c" contained I want it contain "b.c and a.tar.bz2" I don't want to decompress the b.tar.bz2 to achieve this, I try with "cat a.tar.bz2 >>...

6. UNIX and Linux Applications

Download firefox-19.0.2.tar.bz2

Does anyone know a reliable source to download firefox-19.0.2.tar.bz2 from? I would think you be able to download from firefox or mozilla somewhere. I haven't gotten anything useful from my google searches.

7. Ubuntu

Error messages while extracting tar.bz2 in Ubuntu

while extracting a tar.bz2 file using the command tar xjf git.tar.bz2 I received error messages that shows Cannot hard link to and Cannot create symlink to error messages what will be the reason for those error messages.

8. Programming

Aria2c to download and extract. tar.bz2

I am using aria2c to download a .tar.bz2 and trying to extract it in the same command. I can download the file but not extract it. I can also manually extract the tar.bz2., but not in the same command. Thank you :). aria2c -x8 -l log.txt -c -d /xx/xx/xxx --use-head=true --http-user "<user>" ...

9. Shell Programming and Scripting

Bash not removing all .tar.bz2 files after extracting

In the bash below each .tar.bz2 (usually 2) are extracted and then the original .tar.bz2 is removed. However, only one (presumably the first extracted) is being removed, however both are extracted. I am not sure why this is? Thank you :). tar.bz2 folders in /home/cmccabe/Desktop/NGS/API ...

LEARN ABOUT SUSE

virt-tar-in

virt-tar-in(1)						      Virtualization Support						    virt-tar-in(1)

NAME

       virt-tar-in - Unpack a tarball into a virtual machine disk image.

SYNOPSIS

	virt-tar-in -a disk.img data.tar /destination

	virt-tar-in -d domain data.tar /destination

	zcat data.tar.gz | virt-tar-in -d domain - /destination

WARNING

       Using "virt-tar-in" on live virtual machines can be dangerous, potentially causing disk corruption.  The virtual machine must be shut down
       before you use this command.

DESCRIPTION

       "virt-tar-in" unpacks an uncompressed tarball into a virtual machine disk image or named libvirt domain.

       The first parameter is the tar file.  Use "-" to read the tar file from standard input.	The second parameter is the absolute target
       directory to unpack into.

EXAMPLES

       Upload a home directory to a guest:

	virt-tar-in -d MyGuest homes.tar /home

JUST A SHELL SCRIPT WRAPPER AROUND GUESTFISH

       This command is just a simple shell script wrapper around the guestfish(1) "tar-in" command.  For anything more complex than a trivial
       copy, you are probably better off using guestfish directly.

OPTIONS

       Since the shell script just passes options straight to guestfish, read guestfish(1) to see the full list of options.

SEE ALSO

       guestfish(1), virt-cat(1), virt-copy-in(1), virt-copy-out(1), virt-edit(1), virt-make-fs(1), virt-tar-out(1), <http://libguestfs.org/>.

AUTHORS

       Richard W.M. Jones ("rjones at redhat dot com")

COPYRIGHT

       Copyright (C) 2011 Red Hat Inc.	<http://libguestfs.org/>

       This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
       the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

       This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

       You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation,
       Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

libguestfs-1.18.1						    2013-12-07							    virt-tar-in(1)