Sponsored Content
Top Forums Shell Programming and Scripting Fast processing(mv command) of 1 million+ files using find, mv and xargs Post 302789291 by Corona688 on Wednesday 3rd of April 2013 12:14:34 PM
Old 04-03-2013
ls is guaranteed to perform badly here, because it must read the entire directory list and sort their names before it can print. It might bog for minutes or hours until it shows anything.

find doesn't have problems "dealing with" large numbers of files. In a sense find's job is rather simple -- opendir(), readdir(), print if match, loop until done. If it's struggling, that means it either has too much work to do -- finding 300 'good' files out of 1.2 million files you don't care about means scanning through all 1.2 million -- or the filesystem itself is responding slowly.

Small numbers of folders crammed full of millions of files generally perform rather badly, especially when already busy. The filesystem itself, rather than find, may be suffering here.
 

9 More Discussions You Might Find Interesting

1. AIX

command usage on find with xargs and tar

my task : tar up large bunch of files(about 10,000 files) in the current directories that created more than 30 days ago but it come with following error find ./ -ctime +30 | xargs tar rvf test1.tar tar: test1.tar: A file or directory in the path name does not exist. (3 Replies)
Discussion started by: darkrainbow
3 Replies

2. UNIX for Dummies Questions & Answers

use of xargs and prune piping with find command.

Can anyone interpret and tell me the way the below command works? find * -name "*${msgType}" -mtime +${archiveDays} -prune -type f -print 2>/dev/null | xargs rm -f 2> /dev/null Please tell me the usage of prune and xargs in the above command? Looking forward your reply. Thanks in... (1 Reply)
Discussion started by: venkatesht
1 Replies

3. Shell Programming and Scripting

find with xargs to rm found files

I believe what is happening is rm is executing in the script on every directory and on failure of the first it stops although returns status 0. find $HOME -name /directory/filename | xargs -l rm This is the code I use but file remains. I am using sun solaris system which has way limited... (4 Replies)
Discussion started by: Ebodee
4 Replies

4. Solaris

Need to know command to delete more than 3 million files from /var/spool/clientmqueue

Hi I need to delete more than 3 million files from /var/spool/clientmqueue. When I give the following command to delete the files, I get the error # pwd /var/spool/clientmqueue # rm -f * /usr/bin/rm: arg list too long Please tell me how can I delete the files (5 Replies)
Discussion started by: sb200
5 Replies

5. Shell Programming and Scripting

help using find/xargs to apply mp3gain to files

I need to apply mp3gain (album mode) to all mp3 files in a given directory. Each album is in its own directory under /media/data/music/albums for example: /media/data/music/albums/foo /media/data/music/albums/bar /media/data/music/albums/more What needs to happen is: cd... (4 Replies)
Discussion started by: audiophile
4 Replies

6. Shell Programming and Scripting

find numeric duplicates from 300 million lines....

these are numeric ids.. 222932017099186177 222932014385467392 222932017371820032 222932017409556480 I have text file having 300 millions of line as shown above. I want to find duplicates from this file. Please suggest the quicker way.. sort | uniq -d will... (3 Replies)
Discussion started by: pamu
3 Replies

7. UNIX for Dummies Questions & Answers

Deleting a million of files ..

Hi, Which way is faster rm -rf /path/ or find / -name -exec rm {} \; and why? (7 Replies)
Discussion started by: cain82
7 Replies

8. Shell Programming and Scripting

Parallel processing for functions in xargs

I have a script (ksh) which tries to run a function in parallel for performance gains. I am also trying to limit the number of parallel child processes to avoid overloading the system by using a variable to count triggered processes and waiting for completion e.g. do_something () { ... } ... (9 Replies)
Discussion started by: jawsnnn
9 Replies

9. UNIX for Advanced & Expert Users

Zip million files taking 12 hours or more

Hi I have task to zip files based on modified time but they are in millions and it is taking lot of time more than 12 hours and also eating up high cpu is there any other / better way to handle it quickly with less cpu consumptionfind . ! -name \"*.gz\" -mtime +7 -type f | grep -v '/.*/' |... (2 Replies)
Discussion started by: reldb
2 Replies
mdfind(1)						    BSD General Commands Manual 						 mdfind(1)

NAME
mdfind -- finds files matching a given query SYNOPSIS
mdfind [-live] [-count] [-onlyin directory] [-name fileName] query DESCRIPTION
The mdfind command consults the central metadata store and returns a list of files that match the given metadata query. The query can be a string or a query expression. The following options are available: -0 Prints an ASCII NUL character after each result path. This is useful when used in conjunction with xargs -0. -live Causes the mdfind command to provide live-updates to the number of files matching the query. When an update causes the query results to change the number of matches is updated. The find can be cancelled by typing ctrl-C. -count Causes the mdfind command to output the total number of matches, instead of the path to the matching items. -onlyin dir Limit the scope of the search to the directory specified. -name fileName Searches for matching file names only. -literal Force the provided query string to be taken as a literal query string, without interpretation. -interpret Force the provided query string to be interpreted as if the user had typed the string into the Spotlight menu. For example, the string "search" would produce the following query string: (* = search* cdw || kMDItemTextContent = search* cdw) EXAMPLES
The following examples are shown as given to the shell. This returns all files with any metadata attribute value matching the string "image": mdfind image This returns all files that contain "MyFavoriteAuthor" in the kMDItemAuthor metadata attribute: mdfind "kMDItemAuthors == '*MyFavoriteAuthor*'" This returns all files with any metadata attribute value matching the string "skateboard". The find continues to run after gathering the initial results, providing a count of the number of files that match the query. mdfind -live skateboard To get a list of the available attributes for use in constructing queries, see mdimport(1), particularly the -X switch. SEE ALSO
mdimport(1), mdls(1), mdutil(1), xargs(1) Mac OS X June 10, 2004 Mac OS X
All times are GMT -4. The time now is 04:32 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy