Sponsored Content
Top Forums Shell Programming and Scripting Fast processing(mv command) of 1 million+ files using find, mv and xargs Post 302789089 by agentgrecko on Wednesday 3rd of April 2013 04:50:33 AM
Old 04-03-2013
Quote:
Originally Posted by hanson44
Are you sure it's one to two files per second?
Code:
$ ls | wc
   1993    1993   34863

Code:
$ time find . -name "*.*" -type f | xargs -I '{}' mv {} ../xxx
real    0m2.846s
user    0m0.668s
sys     0m2.104s

Code:
$ cd ../xxx
$ ls | wc
   1993    1993   34863

Seems like about 700 files per second. And this is running on kind of a dog of a linux computer, nothing special. Unless your find command is taking days, maybe your operations are going faster than you think. Smilie

At 500 files per second, you could mv a million files in 2000 seconds, about 30 minutes.
H hanson44, yup, around 2 files per sec. Smilie
I have a counter on /destination/dir that executes ls | wc -l every 2 sec just so I could check the progress.
I'm thinking that since /source/dir already contains 1.2 million files (and still receiving more from an auto-dump script), it contributes to the slow processing.
 

9 More Discussions You Might Find Interesting

1. AIX

command usage on find with xargs and tar

my task : tar up large bunch of files(about 10,000 files) in the current directories that created more than 30 days ago but it come with following error find ./ -ctime +30 | xargs tar rvf test1.tar tar: test1.tar: A file or directory in the path name does not exist. (3 Replies)
Discussion started by: darkrainbow
3 Replies

2. UNIX for Dummies Questions & Answers

use of xargs and prune piping with find command.

Can anyone interpret and tell me the way the below command works? find * -name "*${msgType}" -mtime +${archiveDays} -prune -type f -print 2>/dev/null | xargs rm -f 2> /dev/null Please tell me the usage of prune and xargs in the above command? Looking forward your reply. Thanks in... (1 Reply)
Discussion started by: venkatesht
1 Replies

3. Shell Programming and Scripting

find with xargs to rm found files

I believe what is happening is rm is executing in the script on every directory and on failure of the first it stops although returns status 0. find $HOME -name /directory/filename | xargs -l rm This is the code I use but file remains. I am using sun solaris system which has way limited... (4 Replies)
Discussion started by: Ebodee
4 Replies

4. Solaris

Need to know command to delete more than 3 million files from /var/spool/clientmqueue

Hi I need to delete more than 3 million files from /var/spool/clientmqueue. When I give the following command to delete the files, I get the error # pwd /var/spool/clientmqueue # rm -f * /usr/bin/rm: arg list too long Please tell me how can I delete the files (5 Replies)
Discussion started by: sb200
5 Replies

5. Shell Programming and Scripting

help using find/xargs to apply mp3gain to files

I need to apply mp3gain (album mode) to all mp3 files in a given directory. Each album is in its own directory under /media/data/music/albums for example: /media/data/music/albums/foo /media/data/music/albums/bar /media/data/music/albums/more What needs to happen is: cd... (4 Replies)
Discussion started by: audiophile
4 Replies

6. Shell Programming and Scripting

find numeric duplicates from 300 million lines....

these are numeric ids.. 222932017099186177 222932014385467392 222932017371820032 222932017409556480 I have text file having 300 millions of line as shown above. I want to find duplicates from this file. Please suggest the quicker way.. sort | uniq -d will... (3 Replies)
Discussion started by: pamu
3 Replies

7. UNIX for Dummies Questions & Answers

Deleting a million of files ..

Hi, Which way is faster rm -rf /path/ or find / -name -exec rm {} \; and why? (7 Replies)
Discussion started by: cain82
7 Replies

8. Shell Programming and Scripting

Parallel processing for functions in xargs

I have a script (ksh) which tries to run a function in parallel for performance gains. I am also trying to limit the number of parallel child processes to avoid overloading the system by using a variable to count triggered processes and waiting for completion e.g. do_something () { ... } ... (9 Replies)
Discussion started by: jawsnnn
9 Replies

9. UNIX for Advanced & Expert Users

Zip million files taking 12 hours or more

Hi I have task to zip files based on modified time but they are in millions and it is taking lot of time more than 12 hours and also eating up high cpu is there any other / better way to handle it quickly with less cpu consumptionfind . ! -name \"*.gz\" -mtime +7 -type f | grep -v '/.*/' |... (2 Replies)
Discussion started by: reldb
2 Replies
xargs(1)						      General Commands Manual							  xargs(1)

Name
       xargs - construct argument list and execute command

Syntax
       xargs [flags] [ command [initial-arguments] ]

Description
       The command combines fixed initial-arguments with arguments read from standard input to execute a specified command one or more times.  The
       number of arguments read when a command is invoked and how they are combined is determined by the options specified.

       The specified command, (which can be a Shell file) is searched for using ones' $PATH specification.  If command is not specified, /bin/echo
       is used.

       Arguments  read	from  standard	input  are defined as contiguous strings of characters delimited by one or more blanks, tabs, or newlines;
       empty lines are always discarded.  Blanks and tabs can be embedded as part of an argument if they contain an escape character  or  if  they
       are  quoted.   Characters enclosed in quotes (single or double) are taken literally, and the delimiting quotes are removed; a backslash ()
       escapes the next character.

Options
       Each argument list begins with the initial-arguments, followed by arguments read from standard input, with the exception of the -i  option.
       See the description of the -i option for more information.

       The  options -i, -l, and -n determine how arguments are selected when each command is invoked.  If none of these options are specified, the
       initial-arguments are followed by arguments read continuously from standard input until the internal buffer is full; then, command executes
       with  the accumulated arguments.  This process repeats until no arguments exist.  When conflicts arise, such as the -l option used with the
       -n, the last option has precedence. The options values are as follows:

       -lnumber
	     Execute command for each non-empty number lines of arguments from standard input.	When command is invoked for the final time, it has
	     fewer  lines  of  arguments if fewer than a specified number remain.  A line ends with the first newline unless the last character of
	     the line is a blank or a tab; a trailing blank or tab signals continuation through the next non-empty line.   If  number  is  is  not
	     specified, the value 1 is assumed.  The option -x is forced.

       -ireplstr (Insert mode)
	     Execute  command for each line from standard input, taking the entire line as a single argument and inserting it in initial-arguments
	     for each occurrence of replstr.  A maximum of five arguments specified in initial-arguments can contain one  or  more  occurrence	of
	     replstr.	Blanks and tabs at the beginning of each line are discarded.  A constructed arguments cannot exceed 255 characters and the
	     option -x is a forced.  A {|} is assumed for replstr if not specified.

       -nnumber
	     Execute command using as many standard input arguments as possible, up to the specified number arguments  maximum.   Fewer  arguments
	     are used if their total size is greater than size characters, and when the last command is invoked, fewer number of arguments remain.
	     If the option -x is also include, each specified number of arguments must fit in the size limitation, or else xargs terminates execu-
	     tion.

       -t (Trace mode)
	     Echo the command and each constructed argument list to file descriptor 2 prior to their execution.

       -p (Prompt mode)
	     Asks  the user whether or not command should be executed each time command is invoked. Trace mode (-t) is turned on to print the com-
	     mand instance to be executed, followed by a ?... prompt.  A reply of y executes the command; any other response does not invoke  that
	     particular command.

       -x    Causes  the  command xargs to terminate if an argument list is greater than the specified size of characters; the option -x is forced
	     by the options -i and -l.	When the options -i, -l, or -n are included, the total length of all arguments must be within  the  speci-
	     fied size limit.

       -ssize
	     The maximum size of each argument list is set to size characters; size must be a positive integer less than or equal to 470. If -s is
	     not included, 470 is the default.	Note that the character count for size includes one extra character  for  each	argument  and  the
	     count of characters in the command name.

       -eeofstr
	     The  option  eofstr  is  taken  as the logical end-of-file string.  Underscore (_) is assumed for the logical EOF string if -e is not
	     specified.  The value -e without eofstr specified turns off the logical EOF string capability; the  underscore  is  taken	literally.
	     The command xargs reads standard input until either end-of-file or the logical EOF string is encountered.

       The command xargs terminates if it receives a return code of -1 from command or if it cannot execute command.  When command is a Shell pro-
       gram, it should explicitly exit with an appropriate value to avoid returning with -1.  See for more information.

Examples
       The following example moves all files from directory $1 to directory $2 and echoes the move command prior to executing it:

       ls $1 | xargs -i -t mv $1/{} $2/{}

       The following example combines the output of the parenthesized commands onto one line, which is then echoed to the end of file log:

       (logname; date; echo $0 $*) | xargs >>log

       In the next example, the user is prompted to specify which files in the current directory are to be archived.  The first  example  archives
       the files one at a time; the second example archives groups of files:

	 ls | xargs -p -l ar r arch

	 ls | xargs -p -l | xargs ar r arch

       The following example executes diff(1) with successive pairs of arguments originally typed as Shell arguments:

       echo $* | xargs -n2 diff

See Also
       sh(1).

																	  xargs(1)
All times are GMT -4. The time now is 11:27 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy