Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Unexpected Argument list too long error on later os level

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Unexpected Argument list too long error on later os level
# 1  
Old 01-06-2017
Unexpected Argument list too long error on later os level
I have a script on RedHat which runs this:

Code:
ssh -q -t -i $HOME/.ssh/my_key server2 "find ~/toCopy/*data* | xargs sudo mv -f -t /home/files/"

I'm getting:
Code:
sudo: unable to execute /bin/mv: Argument list too long

but the reason I use xargs is to avoid this restriction.

It used to work fine on :
Red Hat Enterprise Linux Server release 5.11 (Tikanga)
but fails on :
Red Hat Enterprise Linux Server release 6.8 (Santiago)

Any ideas, or is there a better command which overcomes this?
# 2  
Old 01-06-2017
You can restrict the number of args used by xargs used for one target call.

Code:
... | xargs -n 50 ....

# 3  
Old 01-06-2017
The old box has:
Code:
xargs --show-limits
Your environment variables take up 1231 bytes
POSIX lower and upper limits on argument length: 2048, 129024
Maximum length of command we could actually use: 127793
Size of command buffer we are actually using: 127793

The new box has:
Code:
Your environment variables take up 2078 bytes
POSIX upper limit on argument length (this system): 2617314
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2615236
Size of command buffer we are actually using: 131072

The base command isn't any bigger, and the target folder is smaller, so I don't get why it doesn't work if Maximum length of command is what it seems to imply.

I've updated my script to use
Code:
-exec sudo mv -f {}

which works (much slower), but I would be interested for opinions on why the other way didn't work - it may be a redhat bug.
Last edited by say170; 01-06-2017 at 01:01 PM..
# 4  
Old 01-06-2017
Quote:
Originally Posted by say170
[..]
I've updated my script to use
Code:
-exec sudo mv -f {}

which works (much slower) [..]
Have you tried using
Code:
-exec sudo mv -f -t /home/files/ {} +

?
# 5  
Old 01-07-2017
That has the same principal problem.
The sudo was not compiled with the system limits. (Update available?)
Did you try stomp's proposal? Gives still good performance.
# 6  
Old 01-08-2017
no. unfortunately I have a variable amount of files to process 3000-5000. This would mean adding a loop to copy in batches which is an ugly solution.
# 7  
Old 01-08-2017
Quote:
Originally Posted by say170
... ... ...

I've updated my script to use
Code:
-exec sudo mv -f {}

which works (much slower), but I would be interested for opinions on why the other way didn't work - it may be a redhat bug.
If, instead of using:
Code:
-exec sudo mv -f {} /home/files/ \;

you use:
Code:
-exec sudo mv -f -t /home/files/ {} +

it should run as fast as (and probably a little bit faster than) what you were getting when xargs was working successfully for you. But, of course, this will only work if find knows the correct limits on exec argument lists.

Quote:
Originally Posted by say170
no. unfortunately I have a variable amount of files to process 3000-5000. This would mean adding a loop to copy in batches which is an ugly solution.
No. Using stomp's suggestion of:
Code:
find ... | xargs -n 50 sudo mv -f -t /home/files/

does not require you to add a loop. This takes exactly the same output from find that you were using before but invokes sudo mv once for every 50 files to be moved instead of trying to fit as many files as it can into one invocation of sudo mv that it thinks it can process with the wrong built-in parameters to determine how many it can use. And, without wasting much time, you could try a considerably higher number as a starting point, for example:
Code:
find ... | xargs -n 1000 sudo mv -f -t /home/files/

and cut the number back to smaller values until you find a number that works if you get the ARGMAX limit exceeded diagnostics with -n 1000.
This User Gave Thanks to Don Cragun For This Post:
say170
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Argument list too long

Hi Team, Here's the situation. I have approximately 300000 to 500000 jpg files in /appl/abcd/work_dir mv /appl/abcd/work_dir /appl/abcd/process_dir The above move command will work if the jpg files count is close to 50000 (not sure). If the count is less this mv command holds good. But if... (14 Replies)
Discussion started by: kmanivan82
14 Replies

2. UNIX for Advanced & Expert Users

Argument list too long w/ sed

Hi all, I am using GNU sed (named gsed under macports) in OSX. I have a directory with a series of files named pool_01.jpg through pool_78802.jpg. I am trying to use this command to rename the files to their checksum + extension. md5sum * | gsed -e 's/\(*\) \(.*\(\..*\)\)$/mv -v \2 \1\3/e' ... (3 Replies)
Discussion started by: openthomas
3 Replies

3. Shell Programming and Scripting

mv : Argument list too long

Hi I am using find command -- find "directory1" -type f | xargs -i mv {} "directory2" to avoid above argument list too long problem. But, issue i am facing is directory1 is having subdirectories due to this i am facing directory traversal problem as i dont want to traverse subdirectories... (9 Replies)
Discussion started by: VSom007
9 Replies

4. Shell Programming and Scripting

grep : Argument list too long

Hi, i am having some trouble with the below command, can some one suggest me the better way to do it. grep -l 'ReturnCode=1' `find $Log -newer /tmp/Failed.tmp -print | xargs ls -ld | egrep SUB | egrep -ve 'MTP' -ve 'ABC' -ve 'DEF' -ve 'JKL' -ve 'XYZ' | awk '{print $9}'` > $Home1 Its... (2 Replies)
Discussion started by: Prateek007
2 Replies

5. Shell Programming and Scripting

Argument list too long!!

Dear Experts, I have a list of 10K files in a directory. I am not able to execute any commands lile ls -lrt, awk, sed, mv, etc........ I wanna execute below command and get the output. How can I achieve it?? Pls help. root# awk -F'|' '$1 == 1' file_20120710* | wc -l /bin/awk: Argument list... (2 Replies)
Discussion started by: Naga06
2 Replies

6. Shell Programming and Scripting

Argument list too long problem

I have a huge set of files (with extension .common) in my directory around 2 million. When I run this script on my Linux with BASH, I get /bin/awk: Argument list too long awk -F'\t' ' NR == FNR { a=NR } NR != FNR { sub(".common", "", FILENAME) print a, FILENAME, $1 } '... (1 Reply)
Discussion started by: shoaibjameel123
1 Replies

7. Shell Programming and Scripting

Argument too long list error

I have a wrote a script which consits of the below line.. Below of this script I'm getting this error "ksh: /usr/bin/ls: arg list too long" The line is log_file_time=`ssh -i $HOME/.ssh/id_rsa -q $i ls -lrt /bp/karthik/test/data/log/$abc*|tail -1|awk '{print $8}'` And $abc alias is as "p |... (1 Reply)
Discussion started by: 22karthikreddy
1 Replies

8. Shell Programming and Scripting

"Argument list too long" error

Hi everyone, I have a problem with my shell script. As a quick overview I need to change a template file 6561 times and copy the file into a new catalogue. Thanks to your forum I have managed to write a script to do so: #!/bin/sh template=$1 for values in {45,165,285}\ {45,165,285}\... (6 Replies)
Discussion started by: mario8eren
6 Replies

9. Shell Programming and Scripting

TAR Files Argument list too long error

Hi, I have a requirement where I need to TAR more than 50K files. Even though I can do TAR successfully on few 100s of files, but whenever Im trying to TAR the entire 50K files, I am getting the error message : Argument List Too Long. Please suggest how can i avoid this error. Im... (2 Replies)
Discussion started by: unx100
2 Replies

10. Shell Programming and Scripting

Argument list too long - Shell error

Trying to tar specific files from a directory causes problems when the number of files is too large. ls ~/logs | wc -l 5928 In the logs directory - I have 5928 files If I want to include all files with today's date - I run the following command tar cf ~/archive/LoadLogs_20060302.tar... (8 Replies)
Discussion started by: dad5119
8 Replies
Login or Register to Ask a Question