"too big" and "not enough memory" errors in shell script

jerardfjay · #1 (**permalink**) 02-05-2009

Hi,

This is odd, however here goes. There are several shell scripts that run in our production environment AIX 595 LPAR m/c, which has sufficient memory 14GB (physical memory) and horsepower 5CPUs. However from time to time we get the following errors in these shell scripts. The time when these happen there is not much activity going on and the errors seem to eventually go away since the scripts which are getting these types of errors are scheduled to run from TWS (maestro) and run all day long. They run just fine during the peak hours of acitivity on the box. Can someone tell me what to check or do for these types of errors.

Code:

 vmstat
System Configuration: lcpu=5 mem=14336MB

Neo · #2 (**permalink**) 02-05-2009

The problem is not the system memory. The limitation is the memory available to read the large files by the commands you are running (like grep, for example).

These must be very large files.

Can you post the size of the files?

jerardfjay · #3 (**permalink**) 02-05-2009

Quote:

Originally Posted by Neo

The problem is not the system memory. The limitation is the memory available to read the large files by the commands you are running (like grep, for example).

These must be very large files.

Can you post the size of the files?

For instance I am posting the typical size of files that got the error on the "rm" command. These are not big at by any means.

Code:

-rw-rw-rw-   1 userid   groupid          266 Feb 05 12:12 AAA.sh_158098.tmp
-rw-rw-rw-   1 userid   groupid          327 Feb 05 12:12 AAA.sh_158098.sql.0
-rw-rw-rw-   1 userid   groupid            6 Feb 05 12:12 AAA.sh_158098.0.AK5File
-rw-rw-rw-   1 userid   groupid           18 Feb 05 12:12 AAA.sh_158098.0.AK2File
-rw-rw-rw-   1 userid   groupid           78 Feb 05 12:12 AAA.sh_158098.0

Neo · #4 (**permalink**) 02-05-2009

You are right. That's odd. I'll think about it and post back.

In the meantime, if anyone has a suggestion, please post.

jerardfjay · #5 (**permalink**) 02-05-2009

Neo,

It is very strange. The frequency of the memory errors are more when compared to the "too big" error. One of the items that IBM support asked us to increase for the "too big" error was the value of ncargs from 512 to 1024. (Initially this value was 6). However this did not have a positive impact.

Here is the individual limits for the userid under which the scripts execute.

Code:

 ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        4194304
memory(kbytes)       unlimited
coredump(blocks)     unlimited
nofiles(descriptors) 8000

bakunin · #6 (**permalink**) 02-06-2009

Hmm, when a command as small as "/usr/bin/rm" does not fit into memory any more (as per your first post here) there is something seriously wrong.

If the problem goes away temporarily with a reboot chances are you have a memory leak problem. To track this sort of problems down is sometimes hard, because of the ephemeral nature of these. You could wait until the problem shows up and then use "ps -Alo vsz<,other options>" to get the "virtual memory footprint" of every process running. See the man page of "ps" for more information about possible options.

Another common source of problems are java processes, which are generally known to be memory hogs. Find out if there are java processes running ("ps -fe | grep java") and if there are some you have a likely cause for your problem.

Further lets examine the overall memory situation of your machine. Please post the output of the following commands:

"svmon -G" (only as root)

"vmstat -v"

"lsps -a"

Also examine the crontabs of all users on the machine. Maybe some memory hog is started regularly and this is whats causing your problems.

I hope this helps.

bakunin

funwithux · #7 (**permalink**) 03-16-2009

I have seen the "too big" error come out when accessing a large number of files, say over 2000. how many files are you trying to process? Also do you have a sample of the scripts that you are trying to run? What else is going on on the server at the time of errors?

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
script running with "ksh" dumping core but not with "sh"	simhe02	HP-UX	9	11-04-2008 08:52 PM
Help req for...shell script to catch "db2 connect" errors ...	frozensmilz	Linux	1	09-23-2008 07:34 PM
Development Releases: Linux Mint 4.0 Beta "Fluxbox", 4.0 Alpha "Debian"	iBot	UNIX and Linux RSS News	0	01-04-2008 03:00 PM
Explain the line "mn_code=`env\|grep "..mn"\|awk -F"=" '{print $2}'`"	Lokesha	UNIX for Dummies Questions & Answers	4	12-20-2007 01:52 AM
No utpmx entry: you must exec "login" from lowest level "shell"	peterpan	UNIX for Dummies Questions & Answers	0	01-18-2006 04:15 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by