Sponsored Content
Top Forums Shell Programming and Scripting Fast processing(mv command) of 1 million+ files using find, mv and xargs Post 302789043 by agentgrecko on Wednesday 3rd of April 2013 02:15:58 AM
Old 04-03-2013
Fast processing(mv command) of 1 million+ files using find, mv and xargs

Hi, I'd like to ask if anybody can help improve my code to move 1 million+ files from a directory to another:

Code:
find /source/dir -name file* -type f | xargs -I '{}' mv {} /destination/dir

I learned this line of code from this forum as well and it works fine. However, file movement is kinda slow; about 1-2 files per second. At this rate, it may take days to move the files. I have not much background yet about xargs, so I was wondering if there could be a faster way to accomplish this process.

Here are some more details:
-OS is HP-UX.
-The files in /source/dir are continually being added.
-Size per file is around 300-1000kb.
-Filename pattern includes YYYYMMDD date (might prove useful for batch processing).
-The mv* command is already encountering "arg list too long," hence the use of find/xargs.
-/source/dir has no sub directories.
-After moving the files, I would later divide/mv then into different dirs corresponding to their YYYYMMDD date.

Hope the above info helps. Any advise would be greatly appreciated as well.

Thank you.
 

9 More Discussions You Might Find Interesting

1. AIX

command usage on find with xargs and tar

my task : tar up large bunch of files(about 10,000 files) in the current directories that created more than 30 days ago but it come with following error find ./ -ctime +30 | xargs tar rvf test1.tar tar: test1.tar: A file or directory in the path name does not exist. (3 Replies)
Discussion started by: darkrainbow
3 Replies

2. UNIX for Dummies Questions & Answers

use of xargs and prune piping with find command.

Can anyone interpret and tell me the way the below command works? find * -name "*${msgType}" -mtime +${archiveDays} -prune -type f -print 2>/dev/null | xargs rm -f 2> /dev/null Please tell me the usage of prune and xargs in the above command? Looking forward your reply. Thanks in... (1 Reply)
Discussion started by: venkatesht
1 Replies

3. Shell Programming and Scripting

find with xargs to rm found files

I believe what is happening is rm is executing in the script on every directory and on failure of the first it stops although returns status 0. find $HOME -name /directory/filename | xargs -l rm This is the code I use but file remains. I am using sun solaris system which has way limited... (4 Replies)
Discussion started by: Ebodee
4 Replies

4. Solaris

Need to know command to delete more than 3 million files from /var/spool/clientmqueue

Hi I need to delete more than 3 million files from /var/spool/clientmqueue. When I give the following command to delete the files, I get the error # pwd /var/spool/clientmqueue # rm -f * /usr/bin/rm: arg list too long Please tell me how can I delete the files (5 Replies)
Discussion started by: sb200
5 Replies

5. Shell Programming and Scripting

help using find/xargs to apply mp3gain to files

I need to apply mp3gain (album mode) to all mp3 files in a given directory. Each album is in its own directory under /media/data/music/albums for example: /media/data/music/albums/foo /media/data/music/albums/bar /media/data/music/albums/more What needs to happen is: cd... (4 Replies)
Discussion started by: audiophile
4 Replies

6. Shell Programming and Scripting

find numeric duplicates from 300 million lines....

these are numeric ids.. 222932017099186177 222932014385467392 222932017371820032 222932017409556480 I have text file having 300 millions of line as shown above. I want to find duplicates from this file. Please suggest the quicker way.. sort | uniq -d will... (3 Replies)
Discussion started by: pamu
3 Replies

7. UNIX for Dummies Questions & Answers

Deleting a million of files ..

Hi, Which way is faster rm -rf /path/ or find / -name -exec rm {} \; and why? (7 Replies)
Discussion started by: cain82
7 Replies

8. Shell Programming and Scripting

Parallel processing for functions in xargs

I have a script (ksh) which tries to run a function in parallel for performance gains. I am also trying to limit the number of parallel child processes to avoid overloading the system by using a variable to count triggered processes and waiting for completion e.g. do_something () { ... } ... (9 Replies)
Discussion started by: jawsnnn
9 Replies

9. UNIX for Advanced & Expert Users

Zip million files taking 12 hours or more

Hi I have task to zip files based on modified time but they are in millions and it is taking lot of time more than 12 hours and also eating up high cpu is there any other / better way to handle it quickly with less cpu consumptionfind . ! -name \"*.gz\" -mtime +7 -type f | grep -v '/.*/' |... (2 Replies)
Discussion started by: reldb
2 Replies
Any::Template::ProcessDir(3pm)				User Contributed Perl Documentation			    Any::Template::ProcessDir(3pm)

NAME
Any::Template::ProcessDir -- Process a directory of templates VERSION
version 0.07 SYNOPSIS
use Any::Template::ProcessDir; # Process templates and generate result files in a single directory # my $pd = Any::Template::ProcessDir->new( dir => '/path/to/dir', process_text => sub { my $template = Any::Template->new( Backend => '...', String => $_[0] ); $template->process({ ... }); } ); $pd->process_dir(); # Process templates and generate result files to a separate directory # my $pd = Any::Template::ProcessDir->new( source_dir => '/path/to/source/dir', dest_dir => '/path/to/dest/dir', process_file => sub { my $file = $_[0]; # do something with $file, return content } ); $pd->process_dir(); DESCRIPTION
Recursively processes a directory of templates, generating a set of result files in the same directory or in a parallel directory. Each file in the source directory may be template-processed, copied, or ignored depending on its pathname. CONSTRUCTOR
Specifying directory/directories o If you want to generate the result files in the same directory as the templates, just specify dir. my $pd = Any::Template::ProcessDir->new( dir => '/path/to/dir', ... ); o If you want to generate the result files in a separate directory from the templates, specify source_dir and dest_dir. my $pd = Any::Template::ProcessDir->new( source_dir => '/path/to/source/dir', source_dir => '/path/to/dest/dir', ... ); Specifying how to process templates process_file A code reference that takes the full template filename and the "Any::Template::ProcessDir" object as arguments, and returns the result string. This can use Any::Template or another method altogether. By default it calls "process_text" on the contents of the file. process_text A code reference that takes the template text and the "Any::Template::ProcessDir" object as arguments, and returns the result string. This can use Any::Template or another method altogether. Optional parameters dir_create_mode Permissions mode to use when creating destination directories. Defaults to 0775. No effect if you are using a single directory. file_create_mode Permissions mode to use when creating destination files. Defaults to 0444 (read-only), so that destination files are not accidentally edited. ignore_files Coderef which takes a full pathname and returns true if the file should be ignored. By default, all files will be considered. readme_filename Name of a README file to generate in the destination directory - defaults to "README". No file will be generated if you pass undef or if you are using a single directory. template_file_suffix Suffix of template files in source directory. Defaults to ".src". This will be removed from the destination file name. Any file in the source directory that does not have this suffix (or "ignore_file_suffix") will simply be copied to the destination. METHODS
process_dir Process the directory. If using multiple directories, the destination directory will be removed completely and recreated, to eliminate any old files from previous processing. SEE ALSO
Any::Template COPYRIGHT AND LICENSE
This software is copyright (c) 2011 by Jonathan Swartz. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. perl v5.14.2 2012-05-14 Any::Template::ProcessDir(3pm)
All times are GMT -4. The time now is 06:57 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy