Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Unexpected Argument list too long error on later os level

Shell Programming and Scripting


Tags
solved, xargs arguments

Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 01-06-2017
say170 say170 is offline
Registered User
 
Join Date: Dec 2011
Last Activity: 10 January 2017, 6:05 AM EST
Posts: 28
Thanks: 6
Thanked 0 Times in 0 Posts
Unexpected Argument list too long error on later os level

I have a script on RedHat which runs this:


Code:
ssh -q -t -i $HOME/.ssh/my_key server2 "find ~/toCopy/*data* | xargs sudo mv -f -t /home/files/"

I'm getting:

Code:
sudo: unable to execute /bin/mv: Argument list too long

but the reason I use xargs is to avoid this restriction.

It used to work fine on :
Red Hat Enterprise Linux Server release 5.11 (Tikanga)
but fails on :
Red Hat Enterprise Linux Server release 6.8 (Santiago)

Any ideas, or is there a better command which overcomes this?
Sponsored Links
    #2  
Old Unix and Linux 01-06-2017
stomp stomp is offline
Registered User
 
Join Date: May 2016
Last Activity: 23 March 2017, 1:13 PM EDT
Posts: 181
Thanks: 9
Thanked 65 Times in 60 Posts
You can restrict the number of args used by xargs used for one target call.


Code:
... | xargs -n 50 ....

Sponsored Links
    #3  
Old Unix and Linux 01-06-2017
say170 say170 is offline
Registered User
 
Join Date: Dec 2011
Last Activity: 10 January 2017, 6:05 AM EST
Posts: 28
Thanks: 6
Thanked 0 Times in 0 Posts
The old box has:

Code:
xargs --show-limits
Your environment variables take up 1231 bytes
POSIX lower and upper limits on argument length: 2048, 129024
Maximum length of command we could actually use: 127793
Size of command buffer we are actually using: 127793

The new box has:

Code:
Your environment variables take up 2078 bytes
POSIX upper limit on argument length (this system): 2617314
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2615236
Size of command buffer we are actually using: 131072

The base command isn't any bigger, and the target folder is smaller, so I don't get why it doesn't work if Maximum length of command is what it seems to imply.

I've updated my script to use
Code:
-exec sudo mv -f {}

which works (much slower), but I would be interested for opinions on why the other way didn't work - it may be a redhat bug.

Last edited by say170; 01-06-2017 at 12:01 PM..
    #4  
Old Unix and Linux 01-06-2017
Scrutinizer's Unix or Linux Image
Scrutinizer Scrutinizer is online now Forum Staff  
Moderator
 
Join Date: Nov 2008
Last Activity: 23 March 2017, 3:56 PM EDT
Location: Amsterdam
Posts: 11,330
Thanks: 479
Thanked 3,244 Times in 2,861 Posts
Quote:
Originally Posted by say170 View Post
[..]
I've updated my script to use
Code:
-exec sudo mv -f {}

which works (much slower) [..]
Have you tried using
Code:
-exec sudo mv -f -t /home/files/ {} +

?
Sponsored Links
    #5  
Old Unix and Linux 01-07-2017
MadeInGermany MadeInGermany is online now Forum Advisor  
Registered User
 
Join Date: May 2012
Last Activity: 23 March 2017, 3:56 PM EDT
Location: Simplicity
Posts: 3,458
Thanks: 266
Thanked 1,140 Times in 1,035 Posts
That has the same principal problem.
The sudo was not compiled with the system limits. (Update available?)
Did you try stomp's proposal? Gives still good performance.
Sponsored Links
    #6  
Old Unix and Linux 01-08-2017
say170 say170 is offline
Registered User
 
Join Date: Dec 2011
Last Activity: 10 January 2017, 6:05 AM EST
Posts: 28
Thanks: 6
Thanked 0 Times in 0 Posts
no. unfortunately I have a variable amount of files to process 3000-5000. This would mean adding a loop to copy in batches which is an ugly solution.
Sponsored Links
    #7  
Old Unix and Linux 01-08-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 23 March 2017, 3:46 PM EDT
Location: San Jose, CA, USA
Posts: 10,110
Thanks: 501
Thanked 3,500 Times in 2,980 Posts
Quote:
Originally Posted by say170 View Post
... ... ...

I've updated my script to use
Code:
-exec sudo mv -f {}

which works (much slower), but I would be interested for opinions on why the other way didn't work - it may be a redhat bug.
If, instead of using:

Code:
-exec sudo mv -f {} /home/files/ \;

you use:

Code:
-exec sudo mv -f -t /home/files/ {} +

it should run as fast as (and probably a little bit faster than) what you were getting when xargs was working successfully for you. But, of course, this will only work if find knows the correct limits on exec argument lists.

Quote:
Originally Posted by say170 View Post
no. unfortunately I have a variable amount of files to process 3000-5000. This would mean adding a loop to copy in batches which is an ugly solution.
No. Using stomp's suggestion of:

Code:
find ... | xargs -n 50 sudo mv -f -t /home/files/

does not require you to add a loop. This takes exactly the same output from find that you were using before but invokes sudo mv once for every 50 files to be moved instead of trying to fit as many files as it can into one invocation of sudo mv that it thinks it can process with the wrong built-in parameters to determine how many it can use. And, without wasting much time, you could try a considerably higher number as a starting point, for example:

Code:
find ... | xargs -n 1000 sudo mv -f -t /home/files/

and cut the number back to smaller values until you find a number that works if you get the ARGMAX limit exceeded diagnostics with -n 1000.
The Following User Says Thank You to Don Cragun For This Useful Post:
say170 (01-10-2017)
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Argument list too long!! Naga06 Shell Programming and Scripting 2 07-11-2012 09:03 AM
Argument too long list error 22karthikreddy Shell Programming and Scripting 1 09-27-2010 01:12 AM
"Argument list too long" error mario8eren Shell Programming and Scripting 6 09-02-2010 05:40 AM
TAR Files Argument list too long error unx100 Shell Programming and Scripting 2 12-28-2009 07:24 AM
Argument list too long - Shell error dad5119 Shell Programming and Scripting 8 03-03-2006 02:57 PM



All times are GMT -4. The time now is 04:21 PM.