UNIX for Dummies Questions & Answers

Every day we back up all files on our system that are older than 7 days, so effectively we do a day's worth at a time.
The way we do this is to issue a find command using mtime +7 - we then loop round and for each result we issue a MV to move the file to a newly created directory. We then TAR the directory and finally compress it to give us a .Z file.

On average we process about 100,000 files per day and this job to do the backup typically takes around 15 hours! Sometimes it can take over 24 hours which causes us problems.

By By adding simple echo statements to the code we have determined that it is the MV commands that are taking the time.

The question therefore is how to improve this.

After a bit of research I tried to pipe the results of the find straight into the TAR using xargs, to avoid doing all the MVs. However in testing this appears to take even longer than before, presumably because TAR is being passed 100,000 files instead of 1 directory containing 100,000 files?

Anyway here is the command -

find . -type f -mtime +7 | xargs -i ksh -c "tar -rf archive.tar" {}; rm {}"

the rm is in there because we need to delete them too.

I am a bit of a newbie so can anyone advise on this or suggest an alternative way?

Thanks in advance,
Tony

Mabye move them to the tar directory daily, then do a tar every week?
Do it in smaller pieces, eg. just 20,000 files instead of 100,000?

Make a little script called fastermove:

Code:

#! /usr/bin/ksh
(($#)) && mv "$@" /destination
exit 0

and then use:
find . -type f -mtime +7 | xargs /path/to/fastermove
(No -i on xargs)

Thanks for that. could you just explain in the script what (($#)) does.

$# in a script stands for the number of args that were sent in to the script from the command line. This does not include the script itself.
i.e. If there are no arguments sent in from command line to the script, the value of $# will be zero.

In the script Perderabo is checking that there is indeed some input to the script before he proceeds to work on the input.

You can check the man page of ksh for more details about the various parameters set by ksh.

Code:

((some_expr)) && some_command

The above bit of code means that if the expression inside the parenthesis evaluates to a non zero value, then execute some_command.

Thanks for the advice on that. I am curious as to how this makes things faster as to all intents and purposes isn't it just doing a mv within the script the same as it would anyway?

You are doing one mv command per file. My script will be invoked by xargs with many files. My guess is that your 15 hour job will take less than an hour with this technique.