Yes, I recall writing a mv ksh script to allow files at the end like xargs:
I wonder if absolute path costs time compared to relative, pawing through more dirs? Having dest on same mount is certainly a winner (no copying).
Last edited by DGPickett; 11-12-2014 at 06:15 PM..
Here's the situation.
I have approximately 300000 to 500000 jpg files in /appl/abcd/work_dir
The above move command will work if the jpg files count is close to 50000 (not sure). If the count is less this mv command holds good. But if the files count is relatively high, the mv command fails by throwing following error message.
but using the following command, it works
Can anyone please let us know how to handle this situation?
Thanks
Krishna
Moderator's Comments:
With 9 formal warnings and infractions, it is obvious that you don't care about the forum rules, and you expect the forum moderators to clean up your posts for you. Enough is enough. Your account has been set to read-only status for three days. Continued refusal to use CODE tags may result in a permanent ban.
There are so many inconsistencies here I'm not sure where to start:
You say you have a directory full of "jpg" files, but the command you say works looks for "vgx" files; not "jpg" files???
You say the command:
fails with an E2BIG error, but that would not happen with this command. Was your command something more like one of the following instead?:
If the command:
works (even though it should give you a syntax error and if that was fixed would only work if you are located in the root directory when you run it), what situation do you need to handle??? If what you really meant was that the above command doesn't work, try:
Adding the cd command and changing the destination in the find -exec clause should let it run a little bit faster (but moving a few hundred thousand files one by one is not going to be fast by any stretch of the imagination. Others have already suggested ways to write a front-end to mv that will allow you to use -exec ... + instead of -exec ... \;.
If you don't have GNU mv, the following should still be pretty efficient:
Create a file named $HOME/bin/shift_mv containing:
and make it executable with:
Then the command:
should be relatively efficient.
(Note that shift_mv does not accept any options; so, if you want to give additional options to mv, you must put them in shift_mv; not in the find -exec clause.)
Safety first - always check exit status of a cd!
Or ---------- Post updated at 06:57 PM ---------- Previous update was at 06:25 PM ----------
I don't see a problem with passing options - should work like other arguments. find -exec might not execute scripts, then you need to explicitly invoke the ksh or sh program.
and the shift_mv can be as small as my embedded path_mv above.
Last edited by MadeInGermany; 11-12-2014 at 08:09 PM..
Reason: source dir corrected
This User Gave Thanks to MadeInGermany For This Post:
I don't see a problem with passing options - should work like other arguments. find -exec might not execute scripts, then you need to explicitly invoke the ksh or sh program.
and the shift_mv can be as small as my embedded path_mv above.
As long as shift_mv is executable and is in one of the directories named in $PATH:
should work just fine without naming a shell to execute it.
You are correct in noting that mv options can be passed to shift_mv; they just won't be treated as options by shift_mv. Having the destination directory operand precede the options violates POSIX Utility Syntax Guideline 9 (All options should precede operands on the command line.), but if we use a SYNOPSIS like:
it is a non-issue. And, of course, the name I suggested (and the path_mv you suggested) also violates POSIX Utility Syntax Guideline 2 (Utility names should include lowercase letters (the lower character classification) and digits only from the portable character set.) by including an underscore.
Is POSIX OK with "mvX OPTIONS DEST_PATH SOURCE_PATH(S)"? I always considered it good form to put the definite (one destination) before the variable (one or more sources).
Hi all,
I am using GNU sed (named gsed under macports) in OSX. I have a directory with a series of files named pool_01.jpg through pool_78802.jpg. I am trying to use this command to rename the files to their checksum + extension.
md5sum * | gsed -e 's/\(*\) \(.*\(\..*\)\)$/mv -v \2 \1\3/e'
... (3 Replies)
Hi I am using find command --
find "directory1" -type f | xargs -i mv {} "directory2"
to avoid above argument list too long problem.
But, issue i am facing is directory1 is having subdirectories due to this i am facing directory traversal problem as i dont want to traverse subdirectories... (9 Replies)
Hi,
i am having some trouble with the below command, can some one suggest me the better way to do it.
grep -l 'ReturnCode=1' `find $Log -newer /tmp/Failed.tmp -print | xargs ls -ld | egrep SUB | egrep -ve 'MTP' -ve 'ABC' -ve 'DEF' -ve 'JKL' -ve 'XYZ' | awk '{print $9}'` > $Home1
Its... (2 Replies)
Dear Experts,
I have a list of 10K files in a directory. I am not able to execute any commands lile ls -lrt, awk, sed, mv, etc........
I wanna execute below command and get the output. How can I achieve it?? Pls help.
root# awk -F'|' '$1 == 1' file_20120710* | wc -l
/bin/awk: Argument list... (2 Replies)
Dear Friends,
The following script processes a 14508 lines log file.
#!/bin/sh
while read line
do
d=`sed 's/* - * \*\/*\/* *\)\] .*/\1/' | tr '/' ' ' | sed 's/\(*\):\(*\)/\1 \2/'`
y=`date -d "${d}" "+%Y%m%d%H%M%S"`
echo "${y}"
done
While running the above script, I am... (4 Replies)
I have a huge set of files (with extension .common) in my directory around 2 million. When I run this script on my Linux with BASH, I get /bin/awk: Argument list too long
awk -F'\t' '
NR == FNR { a=NR }
NR != FNR {
sub(".common", "", FILENAME)
print a, FILENAME, $1
}
'... (1 Reply)
Hi guys
Following command results in
sed -i 's/#/\\#/g' /home/test/sqlstents*
-bash: /bin/sed: Argument list too long
Please help me solve it.. is there any other way i can do this?.. thanks (4 Replies)
I have a wrote a script which consits of the below line.. Below of this script I'm getting this error "ksh: /usr/bin/ls: arg list too long"
The line is
log_file_time=`ssh -i $HOME/.ssh/id_rsa -q $i ls -lrt /bp/karthik/test/data/log/$abc*|tail -1|awk '{print $8}'`
And $abc alias is as "p |... (1 Reply)
Hi
I executed the code
for file in `ls pdb*.ent`
do
new_name=`echo $file | sed 's/^pdb//;s/.ent/.txt/'`
mv $file $new_name
done
Its giving error at ' ls pdb*.ent' argument list too long
i have around 150000 entries
please help
Thank you (6 Replies)
Trying to tar specific files from a directory causes problems when the number of files is too large.
ls ~/logs | wc -l
5928
In the logs directory - I have 5928 files
If I want to include all files with today's date - I run the following command
tar cf ~/archive/LoadLogs_20060302.tar... (8 Replies)