UNIX for Dummies Questions & Answers

Hello everyone,

I've got a question regarding the gzip command.
I regulary use gzip to pack huge ammounts of files.

Is it ok to start 'gzip *' several times in the same directory to parallelize the packing process or can this result in problems, e.g. broken or unpacked files?

My tests till now showed no problems, but perhaps I missed something.

Thanks for your hints,

Basch

In the ssame directory, but of course not the same files... how to you do that, Subdir or using find?
The answer depends on how you do things, and the only serious issue I see could be the impact on performance (it may not be very "Other Users firendly" on a multiuser host...) since it will consume quite some resource

I am pretty novice to UNIX so my approach was/is to issue the following command several times 'gzip * &' in a directory where all contained files should be zipped.
As I understood you, this is not the best way to do it.
So I will have a look at subdir and find to improve my knowledge.

Thanks for your reply, it was really helpfull.

by subdir I ment you would be having separate sub directories to archive... it is not a command...

ah, ok, already got a little bit confused because i found no info about a subdir-command http://unixforums.lg1x8zmax.simplecd...lies/smile.gif find seems also not to be the right tool for the job.

Let me explain with diffrent words, what i want to do:
I have one directory without subdirectories. In this directory i have round about 100 files
Each file is bigger than 1 GB, the biggest file is about 100 GB. The whole directory contains
round about 500 GB of data.
All these files shall be zipped. If i use 'gzip *' once it will take some time to finish zipping all the files.
To speed up the the process I want to use more, maybe 2 or 3, gzip-processes to finish the job.

Now I search for an elegant and safe way to do this.
My first approach was to send the command 'gzip * &' several times.
This seems to work, but I dont feel comfortable with this solution,
because i know it is not elegant and because I don't know if it is safe.
Safe in sense of possible data loss because two gzip-processes try to
zip the same file at the same time or because sth. else goes wrong.

So there are two final questions:
1. Can the problem be solved using the approach I described or are there any dangers?
2. Does anyone know a more sophisticated solution to the problem?

Thanks in advance,

Basch


***Edited on 31.08.09 to correct formating ***

What OS?, what, how many CPU? and memory? and swap?.
Are the swap on the same disk as your data?
Lets say corruption I doubt a little ( but not impossible...) since the most obvious situation:
Trying to zip a file already in treatment would lead gzip to ask you what to do (and so that process would be stopped). What can happen is performance issue but then only you know (is it a multi-user box, a linux PC, or a monolithic server?)

You are right that the gzips will try to do eachother's job with multiple "gzip *"'s. Instead you should carve up the amount of work and give each gzip a limited scope.

You could try:

Code:

#!/usr/bin/ksh
DIR=$1
MAX_PARALLEL=8
parallel=1
for f in $DIR/*; do
  gzip $f &
  (( parallel++ ))
  if (( parallel > MAX_PARALLEL )); then
    wait
    parallel=1
  fi
done
wait

or perhaps use xargs to do the job

Code:

#!/usr/bin/ksh
DIR=$1
setsize=10
ls -1 $DIR/* | xargs -n $setsize | while read workset; do
  gzip $workset &
done
wait

Better yet you could determine the size of the workset on the basis of the amount of files available.

Code:

#!/usr/bin/ksh
DIR=$1
MAX_PARALLEL=8
nroffiles=$(ls $DIR|wc -w)
(( setsize=nroffiles/MAX_PARALLEL ))
ls -1 $DIR/* | xargs -n $setsize | while read workset; do
  gzip $workset&
done
wait

On the command line you could determine the setsize manually. E.g. for 1000 files you would have 125 files per gzip, using 8 gzips so it becomes:

Code:

cd dir
ls -1 * | xargs -n 125 | while read workset; do
  gzip $workset&
done

or

Code:

cd dir
ls -1 * |xargs -P 8 -n 125 gzip

note: I used ls -1 (one) not ls -l