Remember that we're processing a single directory containing 690,000 files. So, we have some constraints...

In theory for i in *.txt should work, but even though no exec is involved, we are still talking about a list of arguments that is probably well over 7.5Mb (and the shell will waste time sorting this list when the order in which the files are processed doesn't matter for this project).

I can't use:

Code:

find . -name '*.txt' ! -name '* *' | "figure out where file should go and move it"

because the behavior of find is undefined if the directory changes while find is reading it.

Invoking sed (or any other utility 690,000 times) to determine the directory to which a file should be moved will take forever. Similarly, invoking mv 690,000 times will take forever. We need to efficiently determine to which directory a list of files should move and move those files in large groups (not individually).

Once I have the list of files to go to a directory, I can use:

Code:

xargs -J 'Arg' mv 'Arg' target_directory

on OS X to minimize the number of needed invocations of mv.

Even if we do this in the shell using entirely shell built-ins to determine the directory to which a file should be moved, echo filename>> listN or printf filename>> listN will still be opening and closing the list files 690,000 times.

There will be more target directories than there are available open files in an awk script on OS X, but we don't know the maximum number of hyphens (nor the number of different values for the number of hyphens) in the 1st word of the 3rd line of these 690,000 files. (We do know that there can be at least 18 hyphens.) I think I can use a pipeline with:

Code:

ls -f | awk "select target directory & create up to 16 lists" | awk "create up to 17 lists"

and easily just read the first 3 lines of the files being processed and just open and close the list files once. The first stage of the above pipeline could also be replaced by the find command mentioned before and that would simplify the 1st awk script in the pipeline. (This could fairly easily be extended to let awk spawn more copies of itself to handle an unlimited number of open list files, but I don't think it will be needed for this project.)

I have a good start on this pipeline, but it will take me a while to finish the code and test it.