Shell Programming and Scripting

HP UNIX version 10.20

I have been using system variable names in some shell scripts in order to automate execution of some test software. I have recently found that there appears to be a restriction with the 'ls' command when listing specific files (e.g. ls *.c). If I pipe the output into wc, I get a return char count value IF it is less than 12,000 (see example below)

ls /home/w9054disk/wright/projects/a318/*.c | wc
(returns)
272 272 9744

This works ok (I then load the output into an array - no problems so far).

However, if I do this on another directory (where the count of files is (say) 20% more) all I get is 'arg list too long'.

Now, I have figured out that it appears to relate to pathname/filename length and the number of entries. Where the magic number of (length of pathname in chars + length of filename in chars) * (number of entries) exceeds 12,000, it does not work. OK, if I go into the directory itself (i.e. no pathname) then in this case it works, BUT, there will come a point where I will go through the magic number again (this is mostly due to the amount of files needing processing).

Does anyone know whether (aka Windows virtual memory limits) there is a user and/or sys admin command I can do (or the sys admin people can do) to increase the limit so that restriction is either moved entirely or at least increased to a higher limit.

This is a real pain at the moment and the system admin people are telling me there is nothing they can do for me.

Thanks for any help, David Wright

You are bumping into LINE_MAX, the maximum length of a line. It is documented on the "limits" manpage. No, an administrator cannot change it.

You are going to need to switch algorithms. Don't try to have all of your filenames in one command line at once.

Perderabo,
Thanks for the reply. I have obviously missed the point somewhere. The LINE_MAX setting here is 2048 chars. How does this figure relate to the 12,000 chars (plus cr's, linefeeds) that appears to be the restriction.
Below are some positive and negative examples. The closest wc value I have achieved so far is 11,810 (without a failure) and I know it fails when it hits 12,044.
Does your reply still stand, or have I missed something.
Unfortunately, source .c/.h filenames being what they are, can only be picked up by using the filename extension. Developers tend not select alphabetically ordered filenames, otherwise I would use a*.c etc.


Rgds David


ok
wright@w9054$ ls projects/$PROJECTPATH/*.[ch].met | wc
272 272 7840
wright@w9054$ pwd
/home/w9061disk/wright
----------------------------
ok
wright@w9054$ ls $PROJECTPATH/*.[ch].met | wc
272 272 7840
wright@w9054$ pwd
/home/w9061disk/wright
----------------------------
ok
wright@w9054$ PROJECTPATH=wright/projects/a318 ; export PROJECTPATH
wright@w9054$ ls $PROJECTPATH/*.[ch].met | wc
272 272 9744
wright@w9054$ pwd
/home/w9061disk
----------------------------
NICHT WAR
wright@w9054$ PROJECTPATH=w9061disk/wright/projects/a318 ; export PROJECTPATH
wright@w9054$ ls $PROJECTPATH/*.[ch].met | wc
/usr/bin/ksh: /usr/bin/ls: arg list too long
0 0 0
wright@w9054$ pwd
/home
wright@w9054$
------------------------------------------------------------------
OK
wright@w9054$ cd w9061disk/
wright@w9054$ PROJECTPATH=wright/projects/a318 ; export PROJECTPATH
wright@w9054$ ls $PROJECTPATH/*.[ch].met | wc
272 272 9744
wright@w9054$ pwd
/home/w9061disk
wright@w9054$
-----------------------------------
NICHT WAR
wright@w9054$ ls w9061disk/wright/projects/a318/*.[ch].met | wc
/usr/bin/ksh: /usr/bin/ls: arg list too long
0 0 0

First, a command line of 2048 is the last command line that is guaranteed to work. No, that doesn't mean that every programmer has carefully ensured that command lines of 2049 will reliably fail. It does mean that careful programmers will not attempt to exceed 2048.

Just because you need to use the .c extention does not mean that you must put all .c file names into a single command line. Here is a script that counts the characters in the filenames of all .c files in the current directory:

Code:

#! /usr/bin/ksh
count=0
ls -f | while read filename ; do
      if [[ $filename = *.c ]] ; then
             ((count=count+${#filename}))
      fi
done
echo count = $count
exit 0

Perderabo,
Thanks for the information you have provided so far, but I have to take this further.
I have pasted something below which appears to contradict everything to date.
If I run the first system env variable script specified (qac441_nh90) - which does nothing more than create sys variables to support an off the shelf application (QAC). Before execution I get what I want, after running I don't. These happens every time and "would appear" to relate to an 'available space' problem.
The current 'work around' to this is for me to run the 'ls' command in directory where the files are located (i.e. I can't use my system variable name). This works a treat for now, but as projects and the number of modules per project get larger, I am sure I will start hitting the 'unknown' ceiling again. I have to assume from the evidence to date that the sys vars are taking up space which prevents the 'ls' command from working correctly.

David

;; This buffer is for notes you don't want to save, and for Lisp evaluation.
;; If you want to create a file, first visit that file with C-x C-f,
;; then enter the text in that file's own buffer.

wright@w9054$ ls *n* | wc
1088 1088 15414
wright@w9054$ ls *n* | wc
1088 1088 15414
wright@w9054$ ls *n* | wc
1088 1088 15414
wright@w9054$ . /home/w9061disk/wright/qac441_nh90
wright@w9054$ . /home/w9061disk/wright/utils/verification.profile
wright@w9054$ ls *n* | wc
/usr/bin/ksh: /usr/bin/ls: arg list too long
0 0 0
wright@w9054$

Thanks for help so far

Hmmm...you seem to be correct. To paraphrase a great American statesman, my previous guess is no longer operative...

If you do a "man 2 exec" and look at the errors that can occur. In particular, the kernel can return I assume, of course, that you will completely ignore the documented limit of 5120 and will experimentally determine the maximum you can achieve today.

So where does this leave you? If your developers develop more programs your script is going to fail. You seem to realize this. I'm not sure that you realize that an OS patch or upgrade could do the same thing.

I must expand my recommendation to you. Not only must you write code that will keep command lines under 2048 bytes, you also need to ensure that the command line plus the environment are less than 5120 bytes. I'm glad we cleared that up!

You're apparently unwilling to consider coding techniques that do not put all file names in one command line. Well, there is another approach. You could segment your project so that there are many small directories instead of one large one. This would render your code safe. But also, large directories slow unix down since they must be searched sequentially.

These really are the only options I see for you. Sorry for the bad news.