Argument list too long


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Argument list too long
# 8  
Old 11-12-2014
It's also wise to use -- on the mv command line just in case of filename(s) starting with hypnen.
# 9  
Old 11-12-2014
Yes, I recall writing a mv ksh script to allow files at the end like xargs:
Code:
#!/usr/bin/ksh
 
opts="$1"
dir="$2"
shift 
shift
mv $opts "$@" "$dir"

I wonder if absolute path costs time compared to relative, pawing through more dirs? Having dest on same mount is certainly a winner (no copying).

Last edited by DGPickett; 11-12-2014 at 06:15 PM..
# 10  
Old 11-12-2014
Yep that works. Can even be an embedded script:
Code:
ls | xargs -n101 sh -c 'dir=$1; shift; echo mv "$@" "$dir"' path_mv /appl/abcd/process_dir -f --

(Remove the echo to really execute!)

Last edited by MadeInGermany; 11-12-2014 at 07:06 PM.. Reason: add -- as Chubler suggested
# 11  
Old 11-12-2014
Quote:
Originally Posted by kmanivan82
Hi Team,

Here's the situation.
I have approximately 300000 to 500000 jpg files in /appl/abcd/work_dir
Code:
mv  /appl/abcd/work_dir /appl/abcd/process_dir

The above move command will work if the jpg files count is close to 50000 (not sure). If the count is less this mv command holds good. But if the files count is relatively high, the mv command fails by throwing following error message.
Code:
ksh :/bin/mv: Argument list too long

but using the following command, it works
Code:
find ./appl/abcd/work_dir -name -type f -name "*.vgx" -exec mv -f {} /appl/abcd/process_dir \;

Can anyone please let us know how to handle this situation?

Thanks
Krishna
Moderator's Comments:
Mod Comment With 9 formal warnings and infractions, it is obvious that you don't care about the forum rules, and you expect the forum moderators to clean up your posts for you. Enough is enough. Your account has been set to read-only status for three days. Continued refusal to use CODE tags may result in a permanent ban.
There are so many inconsistencies here I'm not sure where to start:
You say you have a directory full of "jpg" files, but the command you say works looks for "vgx" files; not "jpg" files???

You say the command:
Code:
mv  /appl/abcd/work_dir /appl/abcd/process_dir

fails with an E2BIG error, but that would not happen with this command. Was your command something more like one of the following instead?:
Code:
mv /appl/abcd/work_dir/*.jp*g /appl/abcd/process_dir
mv /appl/abcd/work_dir/*.vgx /appl/abcd/process_dir[
mv /appl/abcd/work_dir/* /appl/abcd/process_dir

If the command:
Code:
find ./appl/abcd/work_dir -name -type f -name "*.vgx" -exec mv -f {} /appl/abcd/process_dir \;

works (even though it should give you a syntax error and if that was fixed would only work if you are located in the root directory when you run it), what situation do you need to handle??? If what you really meant was that the above command doesn't work, try:
Code:
cd /appl/abcd/process_dir; find /appl/abcd/work_dir -type f -name "*.vgx" -exec mv -f {} . \;

Adding the cd command and changing the destination in the find -exec clause should let it run a little bit faster (but moving a few hundred thousand files one by one is not going to be fast by any stretch of the imagination. Others have already suggested ways to write a front-end to mv that will allow you to use -exec ... + instead of -exec ... \;.

If you don't have GNU mv, the following should still be pretty efficient:
Create a file named $HOME/bin/shift_mv containing:
Code:
#!/bin/ksh
dest="$1"
shift
mv -f -- "$@" "$dest"

and make it executable with:
Code:
chmod +x $HOME/bin/shift_mv

Then the command:
Code:
cd /appl/abcd; find work_dir -type f -name "*.vgx" -exec shift_mv process_dir {} +

should be relatively efficient.
(Note that shift_mv does not accept any options; so, if you want to give additional options to mv, you must put them in shift_mv; not in the find -exec clause.)
# 12  
Old 11-12-2014
Safety first - always check exit status of a cd!
Code:
cd /appl/abcd/process_dir && find ...

Or
Code:
if cd /appl/abcd/process_dir
then
  find ...
  ...
fi

---------- Post updated at 06:57 PM ---------- Previous update was at 06:25 PM ----------

I don't see a problem with passing options - should work like other arguments.
find -exec might not execute scripts, then you need to explicitly invoke the ksh or sh program.
Code:
cd /appl/abcd/process_dir &&
find /appl/abcd/work_dir -type f -name "*.vgx" -exec sh shift_mv . -f -- {} +

and the shift_mv can be as small as my embedded path_mv above.

Last edited by MadeInGermany; 11-12-2014 at 08:09 PM.. Reason: source dir corrected
This User Gave Thanks to MadeInGermany For This Post:
# 13  
Old 11-12-2014
Quote:
Originally Posted by MadeInGermany
... ... ...

I don't see a problem with passing options - should work like other arguments.
find -exec might not execute scripts, then you need to explicitly invoke the ksh or sh program.
Code:
cd /appl/abcd/process_dir &&
find /appl/abcd/work_dir -type f -name "*.vgx" -exec sh shift_mv . -f -- {} +

and the shift_mv can be as small as my embedded path_mv above.
As long as shift_mv is executable and is in one of the directories named in $PATH:
Code:
find ... -exec path_mv ...

should work just fine without naming a shell to execute it.

You are correct in noting that mv options can be passed to shift_mv; they just won't be treated as options by shift_mv. Having the destination directory operand precede the options violates POSIX Utility Syntax Guideline 9 (All options should precede operands on the command line.), but if we use a SYNOPSIS like:
Code:
SYNOPSIS path_mv destination_directory other_mv_arguments...

it is a non-issue. And, of course, the name I suggested (and the path_mv you suggested) also violates POSIX Utility Syntax Guideline 2 (Utility names should include lowercase letters (the lower character classification) and digits only from the portable character set.) by including an underscore.
# 14  
Old 11-21-2014
Is POSIX OK with "mvX OPTIONS DEST_PATH SOURCE_PATH(S)"? I always considered it good form to put the definite (one destination) before the variable (one or more sources).
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Argument list too long w/ sed

Hi all, I am using GNU sed (named gsed under macports) in OSX. I have a directory with a series of files named pool_01.jpg through pool_78802.jpg. I am trying to use this command to rename the files to their checksum + extension. md5sum * | gsed -e 's/\(*\) \(.*\(\..*\)\)$/mv -v \2 \1\3/e' ... (3 Replies)
Discussion started by: openthomas
3 Replies

2. Shell Programming and Scripting

mv : Argument list too long

Hi I am using find command -- find "directory1" -type f | xargs -i mv {} "directory2" to avoid above argument list too long problem. But, issue i am facing is directory1 is having subdirectories due to this i am facing directory traversal problem as i dont want to traverse subdirectories... (9 Replies)
Discussion started by: VSom007
9 Replies

3. Shell Programming and Scripting

grep : Argument list too long

Hi, i am having some trouble with the below command, can some one suggest me the better way to do it. grep -l 'ReturnCode=1' `find $Log -newer /tmp/Failed.tmp -print | xargs ls -ld | egrep SUB | egrep -ve 'MTP' -ve 'ABC' -ve 'DEF' -ve 'JKL' -ve 'XYZ' | awk '{print $9}'` > $Home1 Its... (2 Replies)
Discussion started by: Prateek007
2 Replies

4. Shell Programming and Scripting

Argument list too long!!

Dear Experts, I have a list of 10K files in a directory. I am not able to execute any commands lile ls -lrt, awk, sed, mv, etc........ I wanna execute below command and get the output. How can I achieve it?? Pls help. root# awk -F'|' '$1 == 1' file_20120710* | wc -l /bin/awk: Argument list... (2 Replies)
Discussion started by: Naga06
2 Replies

5. Shell Programming and Scripting

Argument list too long for date command

Dear Friends, The following script processes a 14508 lines log file. #!/bin/sh while read line do d=`sed 's/* - * \*\/*\/* *\)\] .*/\1/' | tr '/' ' ' | sed 's/\(*\):\(*\)/\1 \2/'` y=`date -d "${d}" "+%Y%m%d%H%M%S"` echo "${y}" done While running the above script, I am... (4 Replies)
Discussion started by: tamil.pamaran
4 Replies

6. Shell Programming and Scripting

Argument list too long problem

I have a huge set of files (with extension .common) in my directory around 2 million. When I run this script on my Linux with BASH, I get /bin/awk: Argument list too long awk -F'\t' ' NR == FNR { a=NR } NR != FNR { sub(".common", "", FILENAME) print a, FILENAME, $1 } '... (1 Reply)
Discussion started by: shoaibjameel123
1 Replies

7. UNIX for Dummies Questions & Answers

Argument list too long for Sed command

Hi guys Following command results in sed -i 's/#/\\#/g' /home/test/sqlstents* -bash: /bin/sed: Argument list too long Please help me solve it.. is there any other way i can do this?.. thanks (4 Replies)
Discussion started by: depakjan
4 Replies

8. Shell Programming and Scripting

Argument too long list error

I have a wrote a script which consits of the below line.. Below of this script I'm getting this error "ksh: /usr/bin/ls: arg list too long" The line is log_file_time=`ssh -i $HOME/.ssh/id_rsa -q $i ls -lrt /bp/karthik/test/data/log/$abc*|tail -1|awk '{print $8}'` And $abc alias is as "p |... (1 Reply)
Discussion started by: 22karthikreddy
1 Replies

9. UNIX for Dummies Questions & Answers

Argument list too long - SSH

Hi I executed the code for file in `ls pdb*.ent` do new_name=`echo $file | sed 's/^pdb//;s/.ent/.txt/'` mv $file $new_name done Its giving error at ' ls pdb*.ent' argument list too long i have around 150000 entries please help Thank you (6 Replies)
Discussion started by: empyrean
6 Replies

10. Shell Programming and Scripting

Argument list too long - Shell error

Trying to tar specific files from a directory causes problems when the number of files is too large. ls ~/logs | wc -l 5928 In the logs directory - I have 5928 files If I want to include all files with today's date - I run the following command tar cf ~/archive/LoadLogs_20060302.tar... (8 Replies)
Discussion started by: dad5119
8 Replies
Login or Register to Ask a Question