Sponsored Content
Top Forums Shell Programming and Scripting Speed up extraction od tar.bz2 files using bash Post 302999013 by Corona688 on Monday 12th of June 2017 11:42:20 AM
Old 06-12-2017
At 10 megabytes per second it sounds like there's some room for improvement. But you can't go too crazy or you'll just slow your disk down to uselessness.

This requires the BASH shell, mostly for the ability to do wait "$ONEPARTICULARTHREAD" instead of wait #for everything

Code:
#!/bin/bash

maxproc=2 # Max number of threads.  Suggest 2, or 3 at most
i=0

# Count files
set -- /home/cmccabe/Desktop/NGS/API/*.tar.bz2
FILES="$#"

# Blank $1 $2 ...
set --

let i=1
for FILE in /home/cmccabe/Desktop/NGS/API/*.tar.bz2
do
        printf "(%2d/%2d)\tProcessing %s\n" "$i" "$FILES" "$FILE"
        let i=i+1

        tar -xvjf "$FILE" -C /home/cmccabe/Desktop/NGS/API >/dev/null &

        # Turn $1=pida $2=pidb $3=pidc $4=pidd, into
        #      $1=pida $2=pidb $3=pidc $4=pidd $5=pide
        set -- "$@" $!

        # Shift removes $1 and moves the rest down, so you get
        # $1=pidb $2=pidc $3=pidd $3=pide
        # $# is the number of arguments.
        if [ "$#" -ge $maxproc ]  ; then wait "$1" ; shift; fi
done

This User Gave Thanks to Corona688 For This Post:
 

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

extracting from tar.bz2

hi could any body tell me how to extract .tar.bz2 files i tried using tar but in vain. i found bzip2 in googling but i could not find it on machine unix tru64 please suggest. (1 Reply)
Discussion started by: Raom
1 Replies

2. Shell Programming and Scripting

compare two tar.bz2

Hello, I am using a bash script to archive directories of text files located in ${root}: tar cf ${root}.tar ${root}* bzip2 ${root}.tar I'd like to compare the newly produced archive two.tar.bz2 with the second latest one.tar.bz2. cmp one.tar.bz2 two.tar.bz2 returns one.tar.bz2 two.tar.bz2... (2 Replies)
Discussion started by: JCR
2 Replies

3. Shell Programming and Scripting

Optimize/speed-up perl extraction

Hi, Is there a way I can extract my data faster. You know my data is 1.2 GB text file with 8Million rows with 38 columns/fields. Imagine how huge this is. How I can optimized the data extraction using perl. That is why I'm creating a script to filter only those informations that I need. Is... (3 Replies)
Discussion started by: pinpe
3 Replies

4. Shell Programming and Scripting

How to unpack and install .tar.bz2 library ?

Hi, I am trying to unpack and install .tar.bz2 library. I was told to cd / and than tar -jxvf /source-of-library-file?...tar.bz2 to get files unpacked and installed into / Darius $ pwd / $ $ tar -jxvf /tmp/local/root/ncurses-dev-addon.tar.bz2 ncurses-dev-addon/... (3 Replies)
Discussion started by: jack2
3 Replies

5. Shell Programming and Scripting

Put one tar.bz2 file to another tar.bz2

Hi experts, I have two tar.bz2 file,: a.tar.bz2 and b.tar.bz2 I want to put a.tar.bz2 in to b.tar.bz2 eg: b.tar.bz2 only have one file named "b.c" contained I want it contain "b.c and a.tar.bz2" I don't want to decompress the b.tar.bz2 to achieve this, I try with "cat a.tar.bz2 >>... (1 Reply)
Discussion started by: yanglei_fage
1 Replies

6. UNIX and Linux Applications

Download firefox-19.0.2.tar.bz2

Does anyone know a reliable source to download firefox-19.0.2.tar.bz2 from? I would think you be able to download from firefox or mozilla somewhere. I haven't gotten anything useful from my google searches. (2 Replies)
Discussion started by: cokedude
2 Replies

7. Ubuntu

Error messages while extracting tar.bz2 in Ubuntu

while extracting a tar.bz2 file using the command tar xjf git.tar.bz2 I received error messages that shows Cannot hard link to and Cannot create symlink to error messages what will be the reason for those error messages. (4 Replies)
Discussion started by: saravana krishn
4 Replies

8. Programming

Aria2c to download and extract. tar.bz2

I am using aria2c to download a .tar.bz2 and trying to extract it in the same command. I can download the file but not extract it. I can also manually extract the tar.bz2., but not in the same command. Thank you :). aria2c -x8 -l log.txt -c -d /xx/xx/xxx --use-head=true --http-user "<user>" ... (8 Replies)
Discussion started by: cmccabe
8 Replies

9. Shell Programming and Scripting

Bash not removing all .tar.bz2 files after extracting

In the bash below each .tar.bz2 (usually 2) are extracted and then the original .tar.bz2 is removed. However, only one (presumably the first extracted) is being removed, however both are extracted. I am not sure why this is? Thank you :). tar.bz2 folders in /home/cmccabe/Desktop/NGS/API ... (3 Replies)
Discussion started by: cmccabe
3 Replies
DELUSER(8)						      System Manager's Manual							DELUSER(8)

NAME
deluser, delgroup - remove a user or group from the system SYNOPSIS
deluser [options] [--force] [--remove-home] [--remove-all-files] [--backup] [--backup-to DIR] user deluser --group [options] group delgroup [options] [--only-if-empty] group deluser [options] user group COMMON OPTIONS [--quiet] [--system] [--help] [--version] [--conf FILE] DESCRIPTION
deluser and delgroup remove users and groups from the system according to command line options and configuration information in /etc/deluser.conf and /etc/adduser.conf. They are friendlier front ends to the userdel and groupdel programs, removing the home directory as option or even all files on the system owned by the user to be removed, running a custom script, and other features. deluser and del- group can be run in one of three modes: Remove a normal user If called with one non-option argument and without the --group option, deluser will remove a normal user. By default, deluser will remove the user without removing the home directory, the mail spool or any other files on the system owned by the user. Removing the home directory and mail spool can be achieved using the --remove-home option. The --remove-all-files option removes all files on the system owned by the user. Note that if you activate both options --remove-home will have no effect because all files including the home directory and mail spool are already covered by the --remove-all-files option. If you want to backup all files before deleting them you can activate the --backup option which will create a file username.tar(.gz|.bz2) in the directory specified by the --backup-to option (defaulting to the current working directory). Both the remove and backup options can also be activated for default in the configuration file /etc/deluser.conf. See deluser.conf(5) for details. If you want to remove the root account (uid 0), then use the --force parameter; this may prevent to remove the root user by accident. If the file /usr/local/sbin/deluser.local exists, it will be executed after the user account has been removed in order to do any local cleanup. The arguments passed to deluser.local are: username uid gid home-directory Remove a group If deluser is called with the --group option, or delgroup is called, a group will be removed. Warning: The primary group of an existing user cannot be removed. If the option --only-if-empty is given, the group won't be removed if it has any members left. Remove a user from a specific group If called with two non-option arguments, deluser will remove a user from a specific group. OPTIONS
--conf FILE Use FILE instead of the default files /etc/deluser.conf and /etc/adduser.conf --group Remove a group. This is the default action if the program is invoked as delgroup. --help Display brief instructions. --quiet Suppress progress messages. --system Only delete if user/group is a system user/group. This avoids accidentally deleting non-system users/groups. Additionally, if the user does not exist, no error value is returned. This option is mainly for use in Debian package maintainer scripts. --backup Backup all files contained in the userhome and the mailspool-file to a file named /$user.tar.bz2 or /$user.tar.gz. --backup-to Place the backup files not in / but in the directory specified by this parameter. This implicitly sets --backup also. --remove-home Remove the home directory of the user and its mailspool. If --backup is specified, the files are deleted after having performed the backup. --remove-all-files Remove all files from the system owned by this user. Note: --remove-home does not have an effect any more. If --backup is specified, the files are deleted after having performed the backup. --version Display version and copyright information. RETURN VALUE
0 The action was successfully executed. 1 The user to delete was not a system account. No action was performed. 2 There is no such user. No action was performed. 3 There is no such group. No action was performed. 4 Internal error. No action was performed. 5 The group to delete is not empty. No action was performed. 6 The user does not belong to the specified group. No action was performed. 7 You cannot remove a user from its primary group. No action was performed. 8 The required perl-package 'perl modules' is not installed. This package is required to perform the requested actions. No action was performed. 9 For removing the root account the parameter "--force" is required. No action was performed. FILES
/etc/deluser.conf SEE ALSO
deluser.conf(5), adduser(8), userdel(8), groupdel(8) COPYRIGHT
Copyright (C) 2000 Roland Bauerschmidt. Modifications (C) 2004 Marc Haber and Joerg Hoh. This manpage and the deluser program are based on adduser which is: Copyright (C) 1997, 1998, 1999 Guy Maor. Copyright (C) 1995 Ted Hajek, with a great deal borrowed from the original Debian adduser Copyright (C) 1994 Ian Murdock. deluser is free software; see the GNU General Public Licence version 2 or later for copying conditions. There is no warranty. Debian GNU/Linux Version 3.113+nmu3 DELUSER(8)
All times are GMT -4. The time now is 12:00 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy