UNIX for Dummies Questions & Answers

Ok here is my issue. I have processes which will get screwed up and not run because certain log directories wind up filled too much and a directory will have 100,000 files in it. The only way to fix this is to move files out of this directory until the number of files is small enough that the system will clean it up. No more than 1000 or so files can be there.

This is a long process as I can't just do cp * because I get argument too long. I've been trying to do a

Code:

find * | head -1000 | xargs cp

type of command to simply spew out 1000 files and copy or move them to another folder. Then I could simply !! that until they are all moved. Or loop it somehow. But, I cannot get this to work I do not know what i'm doing wrong here with xargs.

Code:

find * -exec cp {} /tmp/testcopy \;

This does work except that there is no limiting the files there. So i'll just run into the same arguement list too long if the directory is crazy big. I can't limit them by filename or anything like that either. All the names are just crazy numbers with no real meaning. So the only way I can think to limit them si with tail or head. Something like that.

Any ideas would be welcome. I also can't put scripts on boxes or anything. So whatever command I run has to be something I can simply punch into the prompt and execute.

You can give cpio a try:

Code:

find * | cpio -pvdmu /tmp/testcopy

That seems to work. Its copy which isn't ideal because that adds an additional step of having to delete files afterwards as well. Can you think of any way to do this and the result is the files are moved?

Problem with xargs and mv is mv expects the destination on the end. But xargs tacks new arguments onto the end: ls | xargs mv destination will run mv destination file1 file2 file3 file4 which will complain "file4 is not a directory". (Or, if file4 is a directory, will mysteriously vanish everything into it )

Fortunately mv has an option that lets you put the destination before: -t.

Code:

find | xargs mv -t destination

will make it run mv -t destination file1 file2 file3 file4, which is what you need.

find * is redundant, by the way. The way to avoid too many arguments is to not use too many arguments, so don't do globbing at all. Just find by itself will find all files and directories in ./, including ./. You can restrict that further by giving it a specific place to look inside: find ./dir or telling it what kinds of files to find: find -type f (f is files, d is dirs), or certain names: find -iname '*.zip' or telling it a minimum depth to start at: find -mindepth 1 (depth 0 includes ./, depth 1 only includes its contents, etc.) You can keep adding more options to narrow it down to what you need. See man find for full details.

just tells me -t is an illegal option so my mv doesn't seem to support -t

You're not on Linux I take it. Sorry.

Code:

find parameters -exec mv '{}' dest ';'

This isn't as efficient, as it has to run mv hundreds of times.

Yea see that works the problem is I have no way to limit what find picks for files. The directory has 100,000 files with names like.

1276675022.1276721280

So the whole problem is avoiding the arguement list too long problem. When executing copies and moves. Currently I just have to guess and do things like

mv 12766* and hope that it isn't too many files with that name, if it is then I just expand out 1 more digit and try to get as many as I can. Eventually I get them all.

There are no subdirectories or files of other types its all purely log files and how many is entirely dependant on the box and they only get this way in a failed state. So the directory always has 10's of thousands of log files that I have to move.

If i'm able to do a mv command and combine that with head -1000 then that is the only automated way I can avoid arguement list too long problems. Least the only way i've come up with.

My manual system works but its incredibly sucky and time consuming. I have to just keep scrolling through history and slightly altering numbers dozens upon dozens of times.

Its Freebsd with bash BTW.