Shell Programming and Scripting

Hi Team,

Here's the situation.
I have approximately 300000 to 500000 jpg files in /appl/abcd/work_dir

Code:

mv  /appl/abcd/work_dir /appl/abcd/process_dir

The above move command will work if the jpg files count is close to 50000 (not sure). If the count is less this mv command holds good. But if the files count is relatively high, the mv command fails by throwing following error message.

Code:

ksh :/bin/mv: Argument list too long

but using the following command, it works

Code:

find ./appl/abcd/work_dir -name -type f -name "*.vgx" -exec mv -f {} /appl/abcd/process_dir \;

Can anyone please let us know how to handle this situation?

Thanks
Krishna

Moderator's Comments:

With 9 formal warnings and infractions, it is obvious that you don't care about the forum rules, and you expect the forum moderators to clean up your posts for you. Enough is enough. Your account has been set to read-only status for three days. Continued refusal to use CODE tags may result in a permanent ban.

If you get this, use xargs:

Code:

ls | xargs -n101 echo | while read l
do
 mv $l dest
done

Now, with more reading of man xargs, you can find ways to have xargs do the mv, putting the list into the middle of the mv command.

I am presuming your original command was something like:

Code:

mv /appl/abcd/work_dir/*.vgx /appl/abcd/process_dir

Otherwise you would not get that error message.

You are get the message because *.vgx first gets expanded to file1.vgx file2.vgx .... filen.vgx and passed as parameters to the mv command .
Depending on your OS limit at some point that would exceed the ARG_MAX limit or a similar limit for your OS.

How does find set his limit, I wonder? Early xargs was buffer capped at 480 bytes, but that was for way back systems.

Man find says see xargs, man xargs says try for 128K or the argument length limit for exec, less the size of your environment, less 2048 bytes of headroom -- whichever is less; man exec says, well, it gets fuzzy. I wonder if the three are still in sync? https://www.unix.com/man-page/linux/2/execve/

I wrote a few xargs enhancements a fxargs2, where the limit is something like 1K args or 256K of text (how much economy of scale do you want, and why wait to start?), runs are concurrent with subsequent arg collection and if the input blocks, the args accumulated are fired off. Of course, if you are deeper into chaos, there is GNU parallel. However, there will be a grand fight over the target directory inode control!

You could also try:

Code:

find ./appl/abcd/work_dir -name -type f -name "*.vgx" -execdir mv -f -t /appl/abcd/process_dir -- {} +

Which will batch up as many filenames onto the mv command as the commanline size will allow. This is more efficient as you don't end up executing mv once for each file to be moved.

I think there is two times the same problem.

Code:

ls | xargs -i echo mv -f {} /appl/abcd/process_dir

invokes mv for each argument.
It works with GNU mv -t where you can append the arguments:

Code:

ls | xargs -n101 echo mv -f -t /appl/abcd/process_dir

And you need the invokation for each argument with find:

Code:

find ./appl/abcd/work_dir -type f -name "*.vgx" -exec echo mv -f {} /appl/abcd/process_dir \;

Again, with mv -t you can have the {} at the end and the + works.

Code:

find ./appl/abcd/work_dir -type f -name "*.vgx" -exec echo mv -f -t /appl/abcd/process_dir {} +

Remove the echo to really execute!

@DGPickett:
With -exec cmd ... {} + it cuts the argument into chunks that are lower than the limit. But that is not usable here since the + -sign must appear at the end, right after the curly braces..

With -exec cmd ... \; the curly braces can be put anywhere, but the arguments are passed to the command one at a time, which can be slow..


--
@MadeinGermany: thanks for the GNU mv -t option with which the + can be used after all (but GNU mv needs to be present)..