Copying files automatically


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Copying files automatically
# 1  
Old 06-18-2010
Copying files automatically

Ok here is my issue. I have processes which will get screwed up and not run because certain log directories wind up filled too much and a directory will have 100,000 files in it. The only way to fix this is to move files out of this directory until the number of files is small enough that the system will clean it up. No more than 1000 or so files can be there.

This is a long process as I can't just do cp * because I get argument too long. I've been trying to do a
Code:
find * | head -1000 | xargs cp

type of command to simply spew out 1000 files and copy or move them to another folder. Then I could simply !! that until they are all moved. Or loop it somehow. But, I cannot get this to work I do not know what i'm doing wrong here with xargs.

Code:
find * -exec cp {} /tmp/testcopy \;

This does work except that there is no limiting the files there. So i'll just run into the same arguement list too long if the directory is crazy big. I can't limit them by filename or anything like that either. All the names are just crazy numbers with no real meaning. So the only way I can think to limit them si with tail or head. Something like that.

Any ideas would be welcome. I also can't put scripts on boxes or anything. So whatever command I run has to be something I can simply punch into the prompt and execute.
# 2  
Old 06-18-2010
You can give cpio a try:
Code:
find * | cpio -pvdmu /tmp/testcopy

# 3  
Old 06-18-2010
That seems to work. Its copy which isn't ideal because that adds an additional step of having to delete files afterwards as well. Can you think of any way to do this and the result is the files are moved?
# 4  
Old 06-18-2010
Problem with xargs and mv is mv expects the destination on the end. But xargs tacks new arguments onto the end: ls | xargs mv destination will run mv destination file1 file2 file3 file4 which will complain "file4 is not a directory". (Or, if file4 is a directory, will mysteriously vanish everything into it Smilie )

Fortunately mv has an option that lets you put the destination before: -t.

Code:
find | xargs mv -t destination

will make it run mv -t destination file1 file2 file3 file4, which is what you need.

find * is redundant, by the way. The way to avoid too many arguments is to not use too many arguments, so don't do globbing at all. Just find by itself will find all files and directories in ./, including ./. You can restrict that further by giving it a specific place to look inside: find ./dir or telling it what kinds of files to find: find -type f (f is files, d is dirs), or certain names: find -iname '*.zip' or telling it a minimum depth to start at: find -mindepth 1 (depth 0 includes ./, depth 1 only includes its contents, etc.) You can keep adding more options to narrow it down to what you need. See man find for full details.

Last edited by Corona688; 06-18-2010 at 06:13 PM..
# 5  
Old 06-18-2010
just tells me -t is an illegal option so my mv doesn't seem to support -t
# 6  
Old 06-18-2010
You're not on Linux I take it. Sorry.
Code:
find parameters -exec mv '{}' dest ';'

This isn't as efficient, as it has to run mv hundreds of times.
# 7  
Old 06-18-2010
Yea see that works the problem is I have no way to limit what find picks for files. The directory has 100,000 files with names like.

1276675022.1276721280

So the whole problem is avoiding the arguement list too long problem. When executing copies and moves. Currently I just have to guess and do things like

mv 12766* and hope that it isn't too many files with that name, if it is then I just expand out 1 more digit and try to get as many as I can. Eventually I get them all.

There are no subdirectories or files of other types its all purely log files and how many is entirely dependant on the box and they only get this way in a failed state. So the directory always has 10's of thousands of log files that I have to move.

If i'm able to do a mv command and combine that with head -1000 then that is the only automated way I can avoid arguement list too long problems. Least the only way i've come up with.

My manual system works but its incredibly sucky and time consuming. I have to just keep scrolling through history and slightly altering numbers dozens upon dozens of times.

Its Freebsd with bash BTW.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Convert a lot of files in subdirectories automatically

Hi, I have a huge structure of directories and subdirectories contsining some data. The lowest folders contain a file "image.png" which need to be converted to "folder.jpg". But how can I do that for all these files automatically? That's what I alredy have find /path -type f -name... (1 Reply)
Discussion started by: KarlKarpfen
1 Replies

2. Shell Programming and Scripting

Delete log files automatically.

Here is the script I want to run to deleted log files after a certain time: touch /usr/WebSphere/AppServer/profiles/AppSrv01/apps/RSA/logs find /usr/WebSphere/AppServer/profiles/AppSrv01/apps/RSA/logs -atime +120 - exec rm -rf {}\; Exerytime I run it, it throws me the error: find: paths must... (2 Replies)
Discussion started by: lennyz04
2 Replies

3. OS X (Apple)

Moving Files Automatically

I'm looking to find/create a script so that when a download is complete, I can run the script in order for it to automatically move a file such as... 'example.avi' into my videos folder I'm a novice when it comes to scripting, any advice/help would be greatly appreciated! Thanks Andrew (10 Replies)
Discussion started by: abudis
10 Replies

4. Shell Programming and Scripting

Copy files by automatically creating directories

I am in a situation where I have some hundreds of thousands of files in a directory (name of directory: big_directory). Now, working on this directory is extremely slow. I now plan to do this: Copy 1000 files in each of the directories by creating the directories automatically. All files have... (1 Reply)
Discussion started by: shoaibjameel123
1 Replies

5. Shell Programming and Scripting

Files copying - [ Listed files alone. ] - Shell script

Hi All, I am doing this for svn patch making. I got the list of files to make the patch. I have the list in a file with path of all the files. To Do From Directory : /myproject/MainDir To Directory : /myproject/data List of files need to copy is in the file: /myproject/filesList.txt ... (4 Replies)
Discussion started by: linuxadmin
4 Replies

6. Shell Programming and Scripting

Shell script to automatically download files

I am new to shell scripting and need to write a program to copy files that are posted as links on a specific url. I want all the links copied with the same file name and the one posted on the webpage containing the url onto a specific directory. That is the first part. The second part of the script... (2 Replies)
Discussion started by: libertyforall
2 Replies

7. UNIX for Advanced & Expert Users

copying of files by userB, dir & files owned by userA

I am userB and have a dir /temp1 This dir is owned by me. How do I recursively copy files from another users's dir userA? I need to preserve the original user who created files, original group information, original create date, mod date etc. I tried cp -pr /home/userA/* . ... (2 Replies)
Discussion started by: Hangman2
2 Replies

8. UNIX for Dummies Questions & Answers

Automatically copying files from server to local drive?

I would like automate the process of copying some logs files from a server to my local hard drive at a set time each week/day. I don't really know anything about creating and scheduling jobs. Is this something I could setup relatively easily? I would like to automatically copy all the logs... (1 Reply)
Discussion started by: Sepia
1 Replies

9. UNIX for Dummies Questions & Answers

How can I automatically find important files???

how can I automatically check if important files exist in a directory and if not, automatically put the important files where they are needed say, I want to put .bashrc and a dozen other important files like it into every user's directory, how can I do this??? how do I check every user's... (4 Replies)
Discussion started by: TRUEST
4 Replies
Login or Register to Ask a Question

Featured Tech Videos