Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Speed up extraction od tar.bz2 files using bash
Top Forums Shell Programming and Scripting Speed up extraction od tar.bz2 files using bash Post 302999013 by Corona688 on Monday 12th of June 2017 11:42:20 AM
Old 06-12-2017
At 10 megabytes per second it sounds like there's some room for improvement. But you can't go too crazy or you'll just slow your disk down to uselessness.

This requires the BASH shell, mostly for the ability to do wait "$ONEPARTICULARTHREAD" instead of wait #for everything

Code:
#!/bin/bash

maxproc=2 # Max number of threads.  Suggest 2, or 3 at most
i=0

# Count files
set -- /home/cmccabe/Desktop/NGS/API/*.tar.bz2
FILES="$#"

# Blank $1 $2 ...
set --

let i=1
for FILE in /home/cmccabe/Desktop/NGS/API/*.tar.bz2
do
        printf "(%2d/%2d)\tProcessing %s\n" "$i" "$FILES" "$FILE"
        let i=i+1

        tar -xvjf "$FILE" -C /home/cmccabe/Desktop/NGS/API >/dev/null &

        # Turn $1=pida $2=pidb $3=pidc $4=pidd, into
        #      $1=pida $2=pidb $3=pidc $4=pidd $5=pide
        set -- "$@" $!

        # Shift removes $1 and moves the rest down, so you get
        # $1=pidb $2=pidc $3=pidd $3=pide
        # $# is the number of arguments.
        if [ "$#" -ge $maxproc ]  ; then wait "$1" ; shift; fi
done

This User Gave Thanks to Corona688 For This Post:
cmccabe
 

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

extracting from tar.bz2

hi could any body tell me how to extract .tar.bz2 files i tried using tar but in vain. i found bzip2 in googling but i could not find it on machine unix tru64 please suggest. (1 Reply)
Discussion started by: Raom
1 Replies

2. Shell Programming and Scripting

compare two tar.bz2

Hello, I am using a bash script to archive directories of text files located in ${root}: tar cf ${root}.tar ${root}* bzip2 ${root}.tar I'd like to compare the newly produced archive two.tar.bz2 with the second latest one.tar.bz2. cmp one.tar.bz2 two.tar.bz2 returns one.tar.bz2 two.tar.bz2... (2 Replies)
Discussion started by: JCR
2 Replies

3. Shell Programming and Scripting

Optimize/speed-up perl extraction

Hi, Is there a way I can extract my data faster. You know my data is 1.2 GB text file with 8Million rows with 38 columns/fields. Imagine how huge this is. How I can optimized the data extraction using perl. That is why I'm creating a script to filter only those informations that I need. Is... (3 Replies)
Discussion started by: pinpe
3 Replies

4. Shell Programming and Scripting

How to unpack and install .tar.bz2 library ?

Hi, I am trying to unpack and install .tar.bz2 library. I was told to cd / and than tar -jxvf /source-of-library-file?...tar.bz2 to get files unpacked and installed into / Darius $ pwd / $ $ tar -jxvf /tmp/local/root/ncurses-dev-addon.tar.bz2 ncurses-dev-addon/... (3 Replies)
Discussion started by: jack2
3 Replies

5. Shell Programming and Scripting

Put one tar.bz2 file to another tar.bz2

Hi experts, I have two tar.bz2 file,: a.tar.bz2 and b.tar.bz2 I want to put a.tar.bz2 in to b.tar.bz2 eg: b.tar.bz2 only have one file named "b.c" contained I want it contain "b.c and a.tar.bz2" I don't want to decompress the b.tar.bz2 to achieve this, I try with "cat a.tar.bz2 >>... (1 Reply)
Discussion started by: yanglei_fage
1 Replies

6. UNIX and Linux Applications

Download firefox-19.0.2.tar.bz2

Does anyone know a reliable source to download firefox-19.0.2.tar.bz2 from? I would think you be able to download from firefox or mozilla somewhere. I haven't gotten anything useful from my google searches. (2 Replies)
Discussion started by: cokedude
2 Replies

7. Ubuntu

Error messages while extracting tar.bz2 in Ubuntu

while extracting a tar.bz2 file using the command tar xjf git.tar.bz2 I received error messages that shows Cannot hard link to and Cannot create symlink to error messages what will be the reason for those error messages. (4 Replies)
Discussion started by: saravana krishn
4 Replies

8. Programming

Aria2c to download and extract. tar.bz2

I am using aria2c to download a .tar.bz2 and trying to extract it in the same command. I can download the file but not extract it. I can also manually extract the tar.bz2., but not in the same command. Thank you :). aria2c -x8 -l log.txt -c -d /xx/xx/xxx --use-head=true --http-user "<user>" ... (8 Replies)
Discussion started by: cmccabe
8 Replies

9. Shell Programming and Scripting

Bash not removing all .tar.bz2 files after extracting

In the bash below each .tar.bz2 (usually 2) are extracted and then the original .tar.bz2 is removed. However, only one (presumably the first extracted) is being removed, however both are extracted. I am not sure why this is? Thank you :). tar.bz2 folders in /home/cmccabe/Desktop/NGS/API ... (3 Replies)
Discussion started by: cmccabe
3 Replies

LEARN ABOUT DEBIAN

unburden-home-dir

UNBURDEN-HOME-DIR(1)						   User Commands					      UNBURDEN-HOME-DIR(1)

NAME

       unburden-home-dir - unburdens home directories from caches and trashes

SYNOPSIS

       unburden-home-dir [ -n | -u | -f filter ]
       unburden-home-dir ( -h | --help | --version )

DESCRIPTION

       unburden-home-dir  unburdens  the  home	directory from files and directory which cause high I/O or disk usage but are neither important if
       they are lost, e.g. caches or trash directory.

       When being run it moves the files and directories given in the configuration file to a location outside the home directory,  e.g.  /tmp	or
       /scratch, and puts appropriate symbolic links in the home directory instead.

OPTIONS

       -f     just  unburden those directory matched by the given filter (a perl regular expression) -- matches the already unburdened directories
	      if used together with -u.

       -F     Do not check for files in use with lsof before (re)moving files.

       -n     dry run (show what would be done)

       -u     undo (reverse the functionality and put stuff back into the home directory)

       -h, --help
	      show this help

       --version
	      show the program's version

EXAMPLES

       Example configuration files can be found at /usr/share/doc/unburden-home-dir/examples on Debian-based systems and in the etc/ directory	of
       the source tar ball.

FILES

       /etc/unburden-home-dir,	 /etc/unburden-home-dir.list,	~/.unburden-home-dir,  ~/.unburden-home-dir.list,  /etc/default/unburden-home-dir,
       /etc/X11/Xsession.d/95unburden-home-dir

       Read /usr/share/doc/unburden-home-dir/README on debianoid installations or README in the source tar ball for an explanation of these files.

SEE ALSO

       corekeeper (http://openvswitch.org/cgi-bin/gitweb.cgi?p=corekeeper), autotrash(1), agedu(1), bleachbit(1).

       For du(1)-like but more comfortable tools, see ncdu(1) (text-mode), baobab(1) (GNOME), filelight(1) (KDE), xdiskusage(1)  (X  tool  calling
       du(1) itself), or xdu(1) (X tool reading du(1) output from STDIN).

AUTHOR

       Unburden Home Dir is written and maintained by Axel Beckert <beckert@phys.ethz.ch>

LICENSE

       Unburden Home Dir is available under the terms of the GNU General Public License (GPL) version 2 or any later version at your option.

Unburden Home Directory 					     May 2012						      UNBURDEN-HOME-DIR(1)
All times are GMT -4. The time now is 06:47 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy