AIX

When these errors occure, can you just test a grep or wc on one of these files to check if it happens when executing on the shell manually too?

Also maybe you should set up some monitoring of any kind; either self written with vmstat etc. for example or maybe set up nmon.

Here is the output from the svmon command on our box.

Code:

svmon -G
               size      inuse       free        pin    virtual
memory      3670016    3411288     258728     222409     554969
pg space    1572864       1402

               work       pers       clnt      lpage
pin          222409          0          0          0
in use       554989          0    2856299          0

Thx
Jerardfjay

Lets see: The output of "svmon" is in memory pages, which are 4k in AIX. The "size" and "inuse" values tell the physical memory and how of that is used. The machine has ~14GB memory installed (3.5 mio of 4k pages) and uses nearly all of it constantly. That the machine uses all of the physically installed memory is OK and to be expected.

The "virtual" column is the overall memory used by applications. The number is small compared to the number of installed memory and this means that the machine has enough memory for its day-to-day-operation. These figures are statistical in nature and this shows that your memory problems are short peaks of dramatically increased memory demand in a otherwise relatively idle machine.

The one java process you found is IMHO not the problem. If i interpret it correctly it is configured to use 256MB and this should be no big problem.

The output of "vmstat" shows nothing exceptional and the "lpstat" shows you have only 6GB of swap configured. This is a bit on the light side for 14GB of real memory, but otherwise only 1% of the swap is in use - it doesn't seem that you need more right now.

This leaves the question what goes wrong on your machine. You said you experience the problems only in very short timeframes. Start with searching the crontabs of all users you might find one (or several) troublemaker(s) which is (are) called only rarely. (I had such a situation once when a machine was experiencing a severe memory shortage with heavy paging activity every three days. We analyzed the situation and found out that a "mksysb" was responsible for the problem. We moved this mksysb-run to another time with less activity and the problem never happened again.)

I hope this helps.

bakunin

Another idea that is being thrown around is the possible overrun of the heap memory. Since we have 32 OS and applications on this system, are there any known limits for this type of memory. Please advise.

Thx
Jerardfjay

I have seen the "too big" error come out when accessing a large number of files, say over 2000. how many files are you trying to process? Also do you have a sample of the scripts that you are trying to run? What else is going on on the server at the time of errors?