Shell Programming and Scripting

Actually, the operating system definitely matters. A command line length limit has nothing to do with the executables (rm, mv, cp, tar, etc...); it is a byproduct of limits imposed on the system call which creates the process. In situations like these, find and/or xargs are our friends.

This may be of interest to some:
ARG_MAX, maximum length of arguments for a new process

Take care,
Alister

This event was in 2002, and so I have forgotten many of the details. I think the find command also barfed. But it was 3AM, of course, and I must've tried a lot of things before settling on the 'for' script.

Hpux 11.0 was a 32bit OS. And it was on a HP V-class machine which had some very Byzantine I/O constraints. Nevertheless I have almost always had success with the tar command but have had my share of troubles with the others.

And note that I qualified my statement with the word "particularly". We had some Tru64 systems which had absolutely no trouble with this sort of problem. I suspect that most/all 64bit OS's will not demonstrate this kind of problem.

It was not a technical reason that I chose to remove one file at a time. The system in question processed several billion dollars in transaction volume with every batch run. I was expected to 'do something' about the problem rather than actually fix it. You know that it will be a bad day at the office when the Public Affairs Officer shows up in the Data Center at 4AM in her pajamas. At least with all those filenames scrolling up across the screen, I could say that something was being done, even if it was not particularly elegant or technically desirable.

running ./configure on a 64-bit linux system:

Code:

...
checking the maximum length of command line arguments... 1572864
...

So, 1.5 million instead of 64 thousand. Which is a lot, but only 24 times larger, and still not big enough for your absolute worst-case scenario there. The rule of thumb is, if you're using enough arguments enough to even ask "does my OS support it" the only truly safe answer is "no" -- even if it works now it won't scale, and won't be portable. If you have that many already you could always have more, you'll find out your OS' limit in an inconvenient way sooner or later.

I can certainly appreciate needing to make an emergency solution. Four million files!!