Sponsored Content
Full Discussion: gzip parallelized
Top Forums UNIX for Dummies Questions & Answers gzip parallelized Post 302347793 by Basch on Wednesday 26th of August 2009 11:51:59 AM
Old 08-26-2009
Detailed problem description

ah, ok, already got a little bit confused because i found no info about a subdir-command http://unixforums.lg1x8zmax.simplecd...lies/smile.gif find seems also not to be the right tool for the job.

Let me explain with diffrent words, what i want to do:
I have one directory without subdirectories. In this directory i have round about 100 files
Each file is bigger than 1 GB, the biggest file is about 100 GB. The whole directory contains
round about 500 GB of data.
All these files shall be zipped. If i use 'gzip *' once it will take some time to finish zipping all the files.
To speed up the the process I want to use more, maybe 2 or 3, gzip-processes to finish the job.

Now I search for an elegant and safe way to do this.
My first approach was to send the command 'gzip * &' several times.
This seems to work, but I dont feel comfortable with this solution,
because i know it is not elegant and because I don't know if it is safe.
Safe in sense of possible data loss because two gzip-processes try to
zip the same file at the same time or because sth. else goes wrong.

So there are two final questions:
1. Can the problem be solved using the approach I described or are there any dangers?
2. Does anyone know a more sophisticated solution to the problem?

Thanks in advance,

Basch


***Edited on 31.08.09 to correct formating ***

Last edited by Basch; 08-31-2009 at 10:41 AM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

using gzip

Hi, I am trying to unzip a file that I unmounted onto a unix machine from a cd I had burned in a Windows machine. The file I am trying to unzip is a .tar file... it was originally a .tar.gz file because it was zipped using gzip. I have tried: % gzip -d hpux.tar (where hpux.tar is the file... (2 Replies)
Discussion started by: nattie_h
2 Replies

2. UNIX for Dummies Questions & Answers

GZIP help, please!

Gurus, My own stupidity (or ignorance...or both) put me in the situation I am in and I need help please: 1-My shell account (OS: HP UX v11) contains several work directories (/docs, /scripts...) 2-Our sysadmin has implemented aggressive disk quotas so I have to compress the files I put here... (2 Replies)
Discussion started by: alan
2 Replies

3. SCO

gzip

ciao a tutti, premesso che sono un principiante di unix, avrei bisogno di gzip/gunzip e SOPRATTUTTO delle istruzioni (ahimè dettagliatissime, come per un bimbo!) per installarlo... grazie mille, ciao (1 Reply)
Discussion started by: mfran2002
1 Replies

4. UNIX for Dummies Questions & Answers

gzip

Hi All, I have some files which are 01.tar.gz, 02.tar.gz,03.tar.gz ........30.tar.gz. when I want to extract the files I 'm using this command "gzip -dc *.tar.gz | tar -xvf -" but it just uncompress 01.tar.gz how can I uncompress all of them ? thanx alice (2 Replies)
Discussion started by: alisevA3
2 Replies

5. Shell Programming and Scripting

gzip

Hi, I want to gzip files in loop. for i in `ls *.xml`; do gzip $i; done But i am gettin error like "/usr/bin/ls: Arg list too long" Also please tell me how to do it with while loop and also using find and then exec. (7 Replies)
Discussion started by: tushar_tus
7 Replies

6. UNIX for Advanced & Expert Users

gzip vs pipe gzip: produce different file size

Hi All, I have a random test file: test.txt, size: 146 $ ll test.txt $ 146 test.txt Take 1: $ cat test.txt | gzip > test.txt.gz $ ll test.txt.gz $ 124 test.txt.gz Take 2: $ gzip test.txt $ ll test.txt.gz $ 133 test.txt.gz As you can see, gzipping a file and piping into gzip... (1 Reply)
Discussion started by: hanfresco
1 Replies

7. Shell Programming and Scripting

Help with GZIP

Hi Gurus, I have a requirement to zip a file using gzip and ftp it to target server. I am using a gzip script as below. gzip.sh #!/bin/ksh /usr/bin/gzip -9 $1 Filename for gzip.sh is passed by an application program. so the output for ./gzip.sh Test_YYYYMMDDHHMMSS.txt (file name is... (1 Reply)
Discussion started by: PRVARMA
1 Replies

8. Shell Programming and Scripting

gzip

how to zip all log file in a folder expect the latest gzip * ---> will zip all log files but I don't want the latest file to be zipped ex: file1, file2, file3, file4, file5 any single command to gzip all files excpet file5 ? (2 Replies)
Discussion started by: rmann
2 Replies

9. Shell Programming and Scripting

Gzip

if ;then echo "mrnet greater 5000" gzip /var/log/mrnet.log /var/log/mrnet.log.1.gz fi i'm looking if mrnet.log is bigger then 5000 then compress mrnet.log to mrnet.log.1.gz but it won't compress. (3 Replies)
Discussion started by: Froob
3 Replies

10. Programming

Need help with counting within parallelized block - OpenMP/C-Programming

Hello together, I have a question for you. Since a few weeks I am trying to programm a small bruteforce application in C on Ubuntu 14.04.1 using Code::Blocks with gcc-4.8.2. My programm is nothing special. It's just for practise, Without hashing or something like that. For me the focus is on... (11 Replies)
Discussion started by: DaveX
11 Replies
GZFORCE(1)                                                    General Commands Manual                                                   GZFORCE(1)

NAME
gzforce - force a '.gz' extension on all gzip files SYNOPSIS
gzforce [ name ... ] DESCRIPTION
gzforce forces a .gz extension on all gzip files so that gzip will not compress them twice. This can be useful for files with names trun- cated after a file transfer. On systems with a 14 char limitation on file names, the original name is truncated to make room for the .gz suffix. For example, 12345678901234 is renamed to 12345678901.gz. A file name such as foo.tgz is left intact. SEE ALSO
gzip(1), gznew(1), gzmore(1), gzgrep(1), gzdiff(1), gzexe(1) ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +--------------------+-----------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +--------------------+-----------------+ |Availability | SUNWgzip | +--------------------+-----------------+ |Interface Stability | External | +--------------------+-----------------+ NOTES
Source for gzip is available in the SUNWgzipS package. GZFORCE(1)
All times are GMT -4. The time now is 08:02 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy