Shell Programming and Scripting

Hi Guys,

I have about 12000 files in a folder and I want to copy these to another folder. I am using the cp command to do this but it errors out saying cp -- argument list too long.

Is there any way to get around this?

I don't want to do a mv but use only cp.

Thanks

Please post exactly what you typed and the exact error message you received. Knowing you Operating System and Shell would also help.

I use tar to move large directories around...use standard input and output:

Code:

tar cvf - .|(cd $DDIR;tar xf -)

Of course this will copy the entire directory. To only move certain files:

Code:

tar cvf - .$FILESPEC|(cd $DDIR; tar xv - )

This is where $DDDIR uis your destination directory and $FILESPEC is the filename specifications (you can use * to expand filenames).

Sorry to be rude, but I repeat.

Please post exactly what you typed and the exact error message you received. Knowing you Operating System and Shell would also help.

Geekasaurus post can be ignored because is uses erroneous syntax for irrelevant commands.

The cp command does not work for directories with large number of files in them. I don't know what the upper limit is; I could probably find out with a lot of research. Also, the mv and rm commands do not work either under the same circumstance.

I suspect that the upper limit of the number of files that it can work on is 65536. I know that it is greater than 10,000 and less than 100,000. The 65536 is a SWAG number based on 26 years of Unix shell-scripting experience.

A number of years ago, our data warehouse people left more than 4194304 million small files in a single directory. I don't know exactly how many, but they used the entire inode table and its size was set at 4194304 per filesystem (hpux 11.0). I had to reboot the system from CDROM, alter the boot parameters and mount the offending filesystem to the RAM image. Only then could I descend the directory tree and find the overfilled directory. As the OP said, the cp command does not work on directories with that many entries in it. Neither do the rm or mv commands.

I cleaned up the filesystem by removing files. I had to use a for-do-done loop and remove only one file at a time; this took many hours.

The shell being used does not matter. Neither does the OS (particularly). It is the same cp command (or rm, or mv) regardless of the shell.

The tar syntax I quoted is in the man page on my AIX box. I've been using something similar to it for more than 20 years. It works.

Thanks, methyl and Geekasauras.

@methyl,
I did not save the error message and do not have the PC here so I will copy the error message next time I try it on the PC and post it.

@Geekasaurus,
I will try the tar command and see if it works.

It has nothing to do with the rm command. The limit is the size of the buffer that commandline parameters can be fit into. This applies to all commands(except certain shell builtins in certain shells). It is a "swag number" because it's a power of two, which memory is often allocated based on. It's the maximum size of the buffer for commandline arguments per process, and system-dependent. You may have been interested in the xargs command. It converts things read on stdin into commandline arguments, spanning across as many executions as necessary.

Code:

# Delete everything in ./dir_containing_files
find ./dir_containing_files -type f -print0 | xargs --null rm

It works in certain situations, yes. It will probably do what the OP wants. I have my doubts that "$FILESPEC" is working the way you think it is though -- either that or AIX has a very strange implementation of tar...