UNIX for Dummies Questions & Answers

The whole point of my suggestion to use a script was to use the script to rearrange arguments to be passed to a simple command thereby allowing find's -exec primary to exec the underlying UNIX command with a large number of file operands instead of with a single file operand. I tried an experiment using the following commands on my relatively old MacBook Pro running Mac OS X Version 10.7.5, which sets {ARG_MAX} to 262144 bytes (i.e., 256Kb). I used cp as the test command to copy all of the PDF files found in and under my home directory to /tmp/pdfdest. /tmp/pdfdest was a new directory when I started this test, but I did not empty and recreate the directory between tests.

I used the command:

Code:

time find $HOME -name '*.pdf' -exec cp -f {} /tmp/pdfdest \;

three times and ignored the 1st time (which ran a lot slower than the other two as it cleared out the various caches and loaded my home directory's file hierarchy). The two remaining runs averaged 2 minutes 27.45 seconds wall clock time, 0.85 seconds user clock time, and 5.56 seconds system time to copy 881 files by invoking cp 881 times.

Adding the following Korn shell script (named CpDest1st):

Code:

#!/bin/ksh
# Usage: CpDest1st options destdir srcfile...
opts="$1"
dest="$2"
shift 2
exec cp $opts "$@" "$dest"

and using the command:

Code:

time find $HOME -name '*.pdf' -exec CpDest1st -f /tmp/pdfdest {} +

three times and again ignored the first set of results and averaged the other two sets of results. Copying the same 881 files using this method invoking the shell script and the cp utility once each took 2 minutes 3.02 seconds wall time, 0.42 seconds user time, and 3.76 seconds system time.

This isn't a statistically valid comparison and your mileage will vary depending on the value of {ARG_MAX} on your system, the number and sizes of the PDF files being copied, ... . But, it does show that the overhead of using a shell script to reduce the number of invocations of cp (or mv or many other utilities) may actually reduce the elapsed time and the amount of system resources used. In this simple test, using the shell script reduced wall clock time 16%, reduced system time 50%, and reduced user time 32%.

Interesting result. Hmmm. I'm not sure that cp was exec'ed 881 times in the second case; the + sign will collect all file names as parameters, which you allow for using the @ sign in your script. Could you set the -x shell option and show us what happens?

I must not have been clear about what happened.

In the 1st case cp was invoked 881 times to copy 881 files. In the 2nd case, the shell script CpDest1st was invoked 1 time and it invoked cp 1 time to copy the same 881 files. The cp command line invoked was 65190 bytes long. Even though it is just a list of PDF files preceded by cp -f and ending with /Users/dwc/tc1-2008detailed.pdf /Users/dwc/tc1-2008summary.pdf /tmp/pdfdest, I'm not comfortable posting the names of the other 879 PDF files both because this forum is not set up to handle a 65000+ character line length and because the directory structure and file names of files under my home directory is nobody's business but mine.

That's what I thought. If in the first example you invoke cp once with 881 filenames, then you compare apples with apples. find needs the {} to appear just in front of the +, so it does not work out for every command. Fortunately cp has the -t option. So it'd be interesting to invoke it like

Code:

time find $HOME -name '*.pdf' -exec cp -f -t /tmp/pdfdest {} +

Yes, it would. However, cp's -t option is an extension to the standards and is not available on many systems, including OS X (which is the OS I use when testing suggestions I submit to this site).

If your system provides the -t option to cp, use it. It will be faster than using a shell script to rearrange the arguments to move the destination directory to the end of the operand list. If you need a portable solution that will work with all implementations of cp, using a shell script to rearrange the operands may be faster than invoking cp for each file to be moved. Where the tipping point is will vary based on the number of and sizes of the files being copied, and on many factors that will vary from system to system including the number of active users and what they are doing, the value of {ARG_MAX}, the I/O bandwidth, the types of file systems and the underlying hardware used for the source and destination directories, etc.