Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

"Argument list too long" error

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting "Argument list too long" error
# 1  
Old 09-01-2010
"Argument list too long" error
Hi everyone,

I have a problem with my shell script. As a quick overview I need to change a template file 6561 times and copy the file into a new catalogue. Thanks to your forum I have managed to write a script to do so:

Code:
#!/bin/sh

template=$1
for values in {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}; do
    set $values
    A=$1; B=$2; C=$3; D=$4; E=$5; F=$6; G=$7; H=$8
    mkdir "A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
    cp en ./"A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
    cp nf ./"A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
    cp define.sh ./"A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
    cp cosmoprep.sh ./"A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
    sed "s/A\$/$A/;s/B\$/$B/;s/C\$/$C/;s/D\$/$D/;s/E\$/$E/;s/F\$/$F/;s/G\$/$G/;s/H\$/$H/" "template" > ./A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H/"A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
    cd ./A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H
    babel -igzmat "A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H" -oxyz coord.xyz
    source $USELIB/tm510
    x2t coord.xyz > coord
    sh define.sh
    sh cosmoprep.sh
    cd /yki/mount/glauber_home/mario8/Desktop/magister/arvutused/6/
done

Finishing this script takes a lot of time and it went well after I checked it after an hour, but at some point it starts to give errors:

Code:
def.sh: line 7: /bin/mkdir: Argument list too long
def.sh: line 8: /bin/cp: Argument list too long
def.sh: line 9: /bin/cp: Argument list too long
def.sh: line 10: /bin/cp: Argument list too long
def.sh: line 11: /bin/cp: Argument list too long
def.sh: line 12: ./A285-B285-C285-D285-E285-F285-G285-H285/A285-B285-C285-D285-E285-F285-G285-H285: No such file or directory
def.sh: line 13: cd: ./A285-B285-C285-D285-E285-F285-G285-H285: No such file or directory
def.sh: line 14: /usr/local/bin/babel: Argument list too long
/usr/yki/lib/use/tm510: line 30: /bin/hostname: Argument list too long
/usr/yki/science/turbomole-5.10/Config_turbo_env: line 19: /usr/yki/science/turbomole-5.10/scripts/sysname: Argument list too long
def.sh line 16: /usr/yki/science/turbomole-5.10/scripts/x2t: Argument list too long
def.sh: line 17: /bin/sh: Argument list too long
def.sh: line 18: /bin/sh: Argument list too long

I tried to name a folder with the maximum characters my script had and it worked. So I can't see the problem.

Please, help me.

Mario
# 2  
Old 09-01-2010
Yes - but it is the length of the FULL pathname including your directoryname, e.g.,
/path/to/my/list/of/files/./A285-B285-C285-D285-E285-F285-G285-H285
that the system is complaining about.

Code:
grep PATH_MAX /uysr/include/limits.h

will tell you the limit of the number of characters in a fully qualified pathname. Start with that to check your filename problems.
This User Gave Thanks to jim mcnamara For This Post:
mario8eren
# 3  
Old 09-01-2010
Quote:
Originally Posted by jim mcnamara
Yes - but it is the length of the FULL pathname including your directoryname, e.g.,
/path/to/my/list/of/files/./A285-B285-C285-D285-E285-F285-G285-H285
that the system is complaining about.

Code:
grep PATH_MAX /uysr/include/limits.h

will tell you the limit of the number of characters in a fully qualified pathname. Start with that to check your filename problems.
Thank you very much. I tried it, but it didn't find "PATH_MAX" in the file. So I just tried "MAX". I hoped there is an equivalent keyword there, but I didn't find anything I could relate to pathname. I copy the results here, maybe you might notice it:

Code:
mario8@slater (~) >grep PATH_MAX /usr/include/limits.h
mario8@slater (~) >grep MAX /usr/include/limits.h
#define MB_LEN_MAX      16
#  define SCHAR_MAX     127
#  define UCHAR_MAX     255
#   define CHAR_MAX     UCHAR_MAX
#   define CHAR_MAX     SCHAR_MAX
#  define SHRT_MAX      32767
#  define USHRT_MAX     65535
#  define INT_MIN       (-INT_MAX - 1)
#  define INT_MAX       2147483647
#  define UINT_MAX      4294967295U
#   define LONG_MAX     9223372036854775807L
#   define LONG_MAX     2147483647L
#  define LONG_MIN      (-LONG_MAX - 1L)
#   define ULONG_MAX    18446744073709551615UL
#   define ULONG_MAX    4294967295UL
#   define LLONG_MAX    9223372036854775807LL
#   define LLONG_MIN    (-LLONG_MAX - 1LL)
#   define ULLONG_MAX   18446744073709551615ULL
   LLONG_MAX, and ULLONG_MAX.  Instead only the values gcc defined for
#  ifndef LLONG_MAX
#   define LLONG_MAX    LONG_LONG_MAX
#  ifndef ULLONG_MAX
#   define ULLONG_MAX   ULONG_LONG_MAX

Thank you in advance,

Mario

Edit: Maybe I should shorten the script by deleting the letters and only leave the degrees (numbers) in the folder names, since I analyze the results later with another sript and due to that I don't need the letters so badly?
Last edited by mario8eren; 09-01-2010 at 04:39 PM.. Reason: An idea.
# 4  
Old 09-01-2010
We need to know exactly what Operating System you have and the exact version (with all decimal places).
Code:
uname -a

(Blot anything confidential).

Between this post and your previous post about the same script it would appear that the "bash" Shell is keeling over with insufficient environment space and generating spurious or blank command parameters.
Btw. Googling the "mkdir" error message only comes up with Gentoo Linux.

There is excessive repetiton of parameter substitution.
Just saving the common factor for an iteration ready for the next time it is required will save over 400,000 shell parameter substitutions.
e.g.
Code:
INSTANCE="A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"

# 5  
Old 09-02-2010
Quote:
Originally Posted by methyl
We need to know exactly what Operating System you have and the exact version (with all decimal places).
Code:
uname -a

(Blot anything confidential).
I am not sure if this is helpful:

Code:
mario8@yukawa (~) >uname -a
Linux yukawa 2.6.18-6-k7 #1 SMP Fri Feb 19 23:58:00 UTC 2010 i686 GNU/Linux

But I know, that I am using Debian.

Quote:
Between this post and your previous post about the same script it would appear that the "bash" Shell is keeling over with insufficient environment space and generating spurious or blank command parameters.
Btw. Googling the "mkdir" error message only comes up with Gentoo Linux.

There is excessive repetiton of parameter substitution.
Just saving the common factor for an iteration ready for the next time it is required will save over 400,000 shell parameter substitutions.
e.g.
Code:
INSTANCE="A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"

I am not sure if I understood you correct. Are you saying that the script is not working because it has to change too many parameters in one turn and all together it has too many parameters?

And also if you have the time could you provide a link on how to use the command instance exactly, because Googling for it gives me wrong shells and wrong instances Smilie .

Thanks in advance,

Mario

Edit: I just cut the script into little pieces and it seems everything works until the shell has to use "source" command.

Code:
#!/bin/sh

template=$1
for values in {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}; do
    set $values
    A=$1; B=$2; C=$3; D=$4; E=$5; F=$6; G=$7; H=$8
    mkdir "A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
done

These kind of small scripts worked fine.

Even this worked ok:

Code:
#!/bin/sh

template=$1
for values in {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}; do
    set $values
    A=$1; B=$2; C=$3; D=$4; E=$5; F=$6; G=$7; H=$8
    cd ./A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H
    babel -igzmat "A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H" -oxyz coord.xyz
    cd /yki/mount/glauber_home/mario8/Desktop/magister/arvutused/4/
done

When it had to deal with the Turbomole program (tm510) then it worked good at first, but at some point it started to give the "argument list too long" error:

Code:
x2t.sh: line 9: /usr/yki/science/turbomole-5.10/scripts/x2t: Argument list too long
/usr/yki/lib/use/tm510: line 30: /bin/hostname: Argument list too long
/usr/yki/science/turbomole-5.10/Config_turbo_env: line 19: /usr/yki/science/turbomole-5.10/scripts/sysname: Argument list too long
Config_turbo_env: TURBOMOLE script sysname failed!

So I am really confused, because it seems to work at first, but fails at some point. And I removed most of the variables. And I can't understand why the mkdir or babel command worked well and x2t wont work.
Last edited by mario8eren; 09-02-2010 at 05:41 AM.. Reason: Noticed something else while trying new things.
# 6  
Old 09-02-2010
You've got the right approach to diagnose the fault.


Please post x2t.sh .

When executed with "source" it becomes part of def.sh and could well be upsetting def.sh.
This User Gave Thanks to methyl For This Post:
mario8eren
# 7  
Old 09-02-2010
Quote:
Originally Posted by methyl
You've got the right approach to diagnose the fault.


Please post x2t.sh .

When executed with "source" it becomes part of def.sh and could well be upsetting def.sh.
Thank you for the compliment Smilie .

The x2t.sh looks like this:

Code:
#!/bin/sh

template=$1
for values in {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}; do
    set $values
    A=$1; B=$2; C=$3; D=$4; E=$5; F=$6; G=$7; H=$8
    cd ./A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H
    source $USELIB/tm510
    x2t coord.xyz > coord
    cd /yki/mount/glauber_home/mario8/Desktop/magister/arvutused/4/
done

Mario

Edit: It seems you are right, the source command screws it up. I did it like this and it worked:

Code:
mario8@xan01 (test) >use tm510
You can use Turbomole 5.10 now. Type "exit" to leave this mode.
Calculations will be carried out on a single CPU core.
[TM510] mario8@xan01 (test) >sh x2t_2.sh
[TM510] mario8@xan01 (test) >

Code:
#!/bin/sh

template=$1
for values in {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}; do
    set $values
    A=$1; B=$2; C=$3; D=$4; E=$5; F=$6; G=$7; H=$8
    cd ./A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H
    x2t coord.xyz > coord
    cd /yki/mount/glauber_home/mario8/Desktop/magister/arvutused/4/
done

I am going to test the last few lines like that aswell.

Mario

Edit 2: Thank you very much, it worked. "Source was the problem".
Last edited by mario8eren; 09-02-2010 at 09:17 AM.. Reason: New ideas.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Copy of "How to create a long list of directories with mkdir?"

To bakunin and corona688: My result when text in file is ms_ww_546 ms_rrL_99999 ms_nnn_67_756675 is https://www.unix.com/C:\Users\Fejoz\Desktop\ttt.jpg I hope you can see the picture. There is like a "whitespace character" after 2 of the 3 created directories. ---------- Post... (0 Replies)
Discussion started by: setub
0 Replies

2. Shell Programming and Scripting

How to avoid "Too many arguments" error, when passing a long String literal as input to a command?

Hi, I am using awk here. Inside an awk script, I have a variable which contains a very long XML data in string format (500kb). I want to pass this data (as argument) to curl command using system function. But getting Too many arguments error due to length of string data(payloadBlock). I... (4 Replies)
Discussion started by: cool.aquarian
4 Replies

3. Shell Programming and Scripting

Read from "list1" and list matches in "list2"

I want to print any matching IP addresse in List1 with List 2; List 1 List of IP addresses; 161.85.58.210 250.57.15.129 217.23.162.249 74.76.129.101 30.221.177.237 3.147.200.59 170.58.142.64 127.65.109.33 150.167.242.146 223.3.20.186 25.181.180.99 2.55.199.32 (3 Replies)
Discussion started by: lewk
3 Replies

4. Shell Programming and Scripting

Perl "Invalid argument error"

Hi , we have a issue in server, we are running a perl script to connect our clients, but we are not able to connect, every time we are getting the "Invalid argument error" Even i checked all the necessary perl modules are i installed in this server, #create the listen socket my... (2 Replies)
Discussion started by: anishkumarv
2 Replies

5. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

6. Shell Programming and Scripting

Argument too long list error

I have a wrote a script which consits of the below line.. Below of this script I'm getting this error "ksh: /usr/bin/ls: arg list too long" The line is log_file_time=`ssh -i $HOME/.ssh/id_rsa -q $i ls -lrt /bp/karthik/test/data/log/$abc*|tail -1|awk '{print $8}'` And $abc alias is as "p |... (1 Reply)
Discussion started by: 22karthikreddy
1 Replies

7. UNIX for Dummies Questions & Answers

"test: argument expected" error

Hi, No need to say I'm new to unix shell scripting. I have a very simple script that goes this way: for datos in `ls -rt $UNXLOG/26-Jan*` do export arch=`echo $datos |cut -d, -f1` if then export linea1=`grep Debut ${arch}` export horatot=`echo $linea1 |cut -d' ' -f5` ... (7 Replies)
Discussion started by: mvalonso
7 Replies

8. Programming

error "Invalid argument" returned after call sched_setscheduler

the code is below and the was run on Solaris 9. ----------------------------- struct sched_param param; param.sched_priority = 99; if(sched_setscheduler(0, SCHED_RR, &param) == -1) { perror("setting priority"); exit(1); } ------------------------------- after the... (1 Reply)
Discussion started by: robin.zhu
1 Replies

9. Shell Programming and Scripting

Argument list too long - Shell error

Trying to tar specific files from a directory causes problems when the number of files is too large. ls ~/logs | wc -l 5928 In the logs directory - I have 5928 files If I want to include all files with today's date - I run the following command tar cf ~/archive/LoadLogs_20060302.tar... (8 Replies)
Discussion started by: dad5119
8 Replies

10. AIX

How to REMOVE USER that display error --> "Name is too long"

I wish to remove a user from the system... but an error occurr... I try to remove via SMITTY and try to remove via command line with rmuser -p (name of user) and does not works.... The system display that the "Name is too long" There another way to remove a user... or... (1 Reply)
Discussion started by: mgonzal
1 Replies
Login or Register to Ask a Question