Sponsored Content
Top Forums Shell Programming and Scripting Fast processing(mv command) of 1 million+ files using find, mv and xargs Post 302789089 by agentgrecko on Wednesday 3rd of April 2013 04:50:33 AM
Old 04-03-2013
Quote:
Originally Posted by hanson44
Are you sure it's one to two files per second?
Code:
$ ls | wc
   1993    1993   34863

Code:
$ time find . -name "*.*" -type f | xargs -I '{}' mv {} ../xxx
real    0m2.846s
user    0m0.668s
sys     0m2.104s

Code:
$ cd ../xxx
$ ls | wc
   1993    1993   34863

Seems like about 700 files per second. And this is running on kind of a dog of a linux computer, nothing special. Unless your find command is taking days, maybe your operations are going faster than you think. Smilie

At 500 files per second, you could mv a million files in 2000 seconds, about 30 minutes.
H hanson44, yup, around 2 files per sec. Smilie
I have a counter on /destination/dir that executes ls | wc -l every 2 sec just so I could check the progress.
I'm thinking that since /source/dir already contains 1.2 million files (and still receiving more from an auto-dump script), it contributes to the slow processing.
 

9 More Discussions You Might Find Interesting

1. AIX

command usage on find with xargs and tar

my task : tar up large bunch of files(about 10,000 files) in the current directories that created more than 30 days ago but it come with following error find ./ -ctime +30 | xargs tar rvf test1.tar tar: test1.tar: A file or directory in the path name does not exist. (3 Replies)
Discussion started by: darkrainbow
3 Replies

2. UNIX for Dummies Questions & Answers

use of xargs and prune piping with find command.

Can anyone interpret and tell me the way the below command works? find * -name "*${msgType}" -mtime +${archiveDays} -prune -type f -print 2>/dev/null | xargs rm -f 2> /dev/null Please tell me the usage of prune and xargs in the above command? Looking forward your reply. Thanks in... (1 Reply)
Discussion started by: venkatesht
1 Replies

3. Shell Programming and Scripting

find with xargs to rm found files

I believe what is happening is rm is executing in the script on every directory and on failure of the first it stops although returns status 0. find $HOME -name /directory/filename | xargs -l rm This is the code I use but file remains. I am using sun solaris system which has way limited... (4 Replies)
Discussion started by: Ebodee
4 Replies

4. Solaris

Need to know command to delete more than 3 million files from /var/spool/clientmqueue

Hi I need to delete more than 3 million files from /var/spool/clientmqueue. When I give the following command to delete the files, I get the error # pwd /var/spool/clientmqueue # rm -f * /usr/bin/rm: arg list too long Please tell me how can I delete the files (5 Replies)
Discussion started by: sb200
5 Replies

5. Shell Programming and Scripting

help using find/xargs to apply mp3gain to files

I need to apply mp3gain (album mode) to all mp3 files in a given directory. Each album is in its own directory under /media/data/music/albums for example: /media/data/music/albums/foo /media/data/music/albums/bar /media/data/music/albums/more What needs to happen is: cd... (4 Replies)
Discussion started by: audiophile
4 Replies

6. Shell Programming and Scripting

find numeric duplicates from 300 million lines....

these are numeric ids.. 222932017099186177 222932014385467392 222932017371820032 222932017409556480 I have text file having 300 millions of line as shown above. I want to find duplicates from this file. Please suggest the quicker way.. sort | uniq -d will... (3 Replies)
Discussion started by: pamu
3 Replies

7. UNIX for Dummies Questions & Answers

Deleting a million of files ..

Hi, Which way is faster rm -rf /path/ or find / -name -exec rm {} \; and why? (7 Replies)
Discussion started by: cain82
7 Replies

8. Shell Programming and Scripting

Parallel processing for functions in xargs

I have a script (ksh) which tries to run a function in parallel for performance gains. I am also trying to limit the number of parallel child processes to avoid overloading the system by using a variable to count triggered processes and waiting for completion e.g. do_something () { ... } ... (9 Replies)
Discussion started by: jawsnnn
9 Replies

9. UNIX for Advanced & Expert Users

Zip million files taking 12 hours or more

Hi I have task to zip files based on modified time but they are in millions and it is taking lot of time more than 12 hours and also eating up high cpu is there any other / better way to handle it quickly with less cpu consumptionfind . ! -name \"*.gz\" -mtime +7 -type f | grep -v '/.*/' |... (2 Replies)
Discussion started by: reldb
2 Replies
FIND(1) 						      General Commands Manual							   FIND(1)

NAME
find - find files meeting a given condition SYNOPSIS
find directory expression EXAMPLES
find / -name a.out -print # Print all a.out paths find /usr/ast ! -newer f -ok rm {} ; # Ask before removing find /usr -size +20 -exec mv {} /big ; # move files > 20 blks find / -name a.out -o -name '*.o' -exec rm {}; # 2 conds DESCRIPTION
Find descends the file tree starting at the given directory checking each file in that directory and its subdirectories against a predi- cate. If the predicate is true, an action is taken. The predicates may be connected by -a (Boolean and), -o (Boolean or) and ! (Boolean negation). Each predicate is true under the conditions specified below. The integer n may also be +n to mean any value greater than n, -n to mean any value less than n, or just n for exactly n. -name s true if current filename is s (include shell wild cards) -size n true if file size is n blocks -inum n true if the current file's i-node number is n -mtime ntrue if modification time relative to today (in days) is n -links ntrue if the number of links to the file is n -newer ftrue if the file is newer than f -perm n true if the file's permission bits = n (n is in octal) -user u true if the uid = u (a numerical value, not a login name) -group gtrue if the gid = g (a numerical value, not a group name) -type x where x is bcdfug (block, char, dir, regular file, setuid, setgid) -xdev do not cross devices to search mounted file systems Following the expression can be one of the following, telling what to do when a file is found: -print print the file name on standard output -exec execute a MINIX command, {} stands for the file name -ok prompts before executing the command SEE ALSO
test(1), xargs(1). FIND(1)
All times are GMT -4. The time now is 01:53 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy