Sponsored Content
Top Forums Shell Programming and Scripting Fast processing(mv command) of 1 million+ files using find, mv and xargs Post 302789101 by hanson44 on Wednesday 3rd of April 2013 05:11:34 AM
Old 04-03-2013
Yes, perhaps the find command is the bottleneck. Maybe it has a hard time "dealing with" so many files.

What happens if you run the find command for a minute (forget about the mv part for the time being), saving the output from find to a file, and see how many lines accumulate in the file?

If there are perhaps 120 lines after a minute (two per second), then find is the bottleneck. If there are tens of thousands of lines, then it's still a mystery.

Is there any chance you could just use 'ls' instead of find?
 

9 More Discussions You Might Find Interesting

1. AIX

command usage on find with xargs and tar

my task : tar up large bunch of files(about 10,000 files) in the current directories that created more than 30 days ago but it come with following error find ./ -ctime +30 | xargs tar rvf test1.tar tar: test1.tar: A file or directory in the path name does not exist. (3 Replies)
Discussion started by: darkrainbow
3 Replies

2. UNIX for Dummies Questions & Answers

use of xargs and prune piping with find command.

Can anyone interpret and tell me the way the below command works? find * -name "*${msgType}" -mtime +${archiveDays} -prune -type f -print 2>/dev/null | xargs rm -f 2> /dev/null Please tell me the usage of prune and xargs in the above command? Looking forward your reply. Thanks in... (1 Reply)
Discussion started by: venkatesht
1 Replies

3. Shell Programming and Scripting

find with xargs to rm found files

I believe what is happening is rm is executing in the script on every directory and on failure of the first it stops although returns status 0. find $HOME -name /directory/filename | xargs -l rm This is the code I use but file remains. I am using sun solaris system which has way limited... (4 Replies)
Discussion started by: Ebodee
4 Replies

4. Solaris

Need to know command to delete more than 3 million files from /var/spool/clientmqueue

Hi I need to delete more than 3 million files from /var/spool/clientmqueue. When I give the following command to delete the files, I get the error # pwd /var/spool/clientmqueue # rm -f * /usr/bin/rm: arg list too long Please tell me how can I delete the files (5 Replies)
Discussion started by: sb200
5 Replies

5. Shell Programming and Scripting

help using find/xargs to apply mp3gain to files

I need to apply mp3gain (album mode) to all mp3 files in a given directory. Each album is in its own directory under /media/data/music/albums for example: /media/data/music/albums/foo /media/data/music/albums/bar /media/data/music/albums/more What needs to happen is: cd... (4 Replies)
Discussion started by: audiophile
4 Replies

6. Shell Programming and Scripting

find numeric duplicates from 300 million lines....

these are numeric ids.. 222932017099186177 222932014385467392 222932017371820032 222932017409556480 I have text file having 300 millions of line as shown above. I want to find duplicates from this file. Please suggest the quicker way.. sort | uniq -d will... (3 Replies)
Discussion started by: pamu
3 Replies

7. UNIX for Dummies Questions & Answers

Deleting a million of files ..

Hi, Which way is faster rm -rf /path/ or find / -name -exec rm {} \; and why? (7 Replies)
Discussion started by: cain82
7 Replies

8. Shell Programming and Scripting

Parallel processing for functions in xargs

I have a script (ksh) which tries to run a function in parallel for performance gains. I am also trying to limit the number of parallel child processes to avoid overloading the system by using a variable to count triggered processes and waiting for completion e.g. do_something () { ... } ... (9 Replies)
Discussion started by: jawsnnn
9 Replies

9. UNIX for Advanced & Expert Users

Zip million files taking 12 hours or more

Hi I have task to zip files based on modified time but they are in millions and it is taking lot of time more than 12 hours and also eating up high cpu is there any other / better way to handle it quickly with less cpu consumptionfind . ! -name \"*.gz\" -mtime +7 -type f | grep -v '/.*/' |... (2 Replies)
Discussion started by: reldb
2 Replies
lndir(1X)																 lndir(1X)

NAME
lndir - create a shadow directory of symbolic links to another directory tree SYNOPSIS
lndir fromdir [todir] DESCRIPTION
lndir makes a shadow copy todir of a directory tree fromdir, except that the shadow is not populated with real files but instead with sym- bolic links pointing at the real files in the fromdir directory tree. This is usually useful for maintaining source code for different machine architectures. You create a shadow directory containing links to the real source which you will have usually NFS mounted from a machine of a different architecture, and then recompile it. The object files will be in the shadow directory, while the source files in the shadow directory are just symlinks to the real files. This has the advantage that if you update the source, you need not propagate the change to the other architectures by hand, since all source in shadow directories are symlinks to the real thing: just cd to the shadow directory and recompile. The todir argument is optional and defaults to the current directory. The fromdir argument may be relative (e.g., ../src) and is relative to todir (not the current directory). Note that RCS, SCCS, and CVS.adm directories are not shadowed. Note also that if you add files, you must run lndir again. Deleting files is difficult because the symlinks will point to places that no longer exist. BUGS
The patch routine needs to be able to change the files. You should never run patch from a shadow directory. Use a command like the following to clear out all files before you can relink (if the fromdir has been moved, for instance): find todir -type l -print | xargs rm The following command will find all files that are not directories: find . ! -type d -print lndir(1X)
All times are GMT -4. The time now is 05:24 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy