Shell Programming and Scripting

I posted last week about how the find command (known to be slow to begin with), is slowing down by 75x on a windows remote share.

Do awk or Perl have the capability to find files (pretty sure the answer for Perl is yes). I want to duplicate find "$dataDir" -type d -name '*.aps' (recursive search for directories with a certain name returning full path).

Mike

I doubt seriously you'll be able to magically speed things up. Find isn't a slouch. There's only so "fast" you can make an exhaustive search. Now... if you can change the logic of whatever you are doing so that there is a way to do a fast lookup of the file, then fine...

(e.g. whenever I create a *.aps I create a name-value pair in my high speed indexable database... or something else)

The problem is the speed it takes to read 100,000 directory entries on the remote share in question. The language in question, be it C, Perl, Java, or Python, does not matter. They all use the same system calls.

I agree with cjcox there is no magic here...plus you are searching on a remote filesystem.

Depending upon your os (i.e. Redhat) you may want to use the locate, mlocate or slocate command. See the man pages for locate, slocate, mlocate and updatedb. This facility creates an index for finding files.

I know the remote share is responsible for a good part of the overhead, but I am accostomed to finding 3-4x differences in speed trying the same thing in sed, awk, perl, etc. When searching for a faster find on line there are some enticing mentions of perl being particuarly fast/efficient.

Even a 2-3x improvement would help tramendously.

Mike

Computers do not work that way.

A slow system call is slow in any language, and the slower it is, the less there is to be gained by 'optimizing' it.

Suppose your program is spending 98% of its time waiting for NFS and 2% of its time actually running. If you find a 200% faster proram, it will be spending 1% of its time actually running and 99% of its time waiting on NFS with theoretical a speed gain of 1% and a realistic speed gain of absolutely zip.

This is also why you can't turbo charge a slow disk with a fast program. No matter how fast your program is, the underlying I/O can't actually move faster.

You might be able to parallelize it, but only to a point.

Perhaps your network connection or NFS can be fine-tuned? That's beyond my expertise, though.

P.S. The find command is not 'known to be slow', certainly not slower than any other file tree walker I know. If you don't understand why it's 'slow' when used on huge file trees, you don't actually know what it's doing.

The script runs side by side with an extremely un-user-friendly (and expensive) data aquisition software allowing the techs to edit comments after the fact and renename runs (requires editing an xml file, renaming 6 data files, and renaming the *.aps file). It needs to update file and folder lists in real time. When in share drive mode, it is for editing after the fact and it doesn't. I just added code to skip the find command, so the old result is parsed unless the user hits "r" for reset if running in remote mode. You don't need constant updating when you are not running along side the data acquisition software.

I understand how bottlenecks work but am not 100% certain that NFS is the bottleneck.

Mike