"too big" and "not enough memory" errors in shell script


 
Thread Tools Search this Thread
Operating Systems AIX "too big" and "not enough memory" errors in shell script
# 1  
Old 02-05-2009
"too big" and "not enough memory" errors in shell script

Hi,

This is odd, however here goes. There are several shell scripts that run in our production environment AIX 595 LPAR m/c, which has sufficient memory 14GB (physical memory) and horsepower 5CPUs. However from time to time we get the following errors in these shell scripts. The time when these happen there is not much activity going on and the errors seem to eventually go away since the scripts which are getting these types of errors are scheduled to run from TWS (maestro) and run all day long. They run just fine during the peak hours of acitivity on the box. Can someone tell me what to check or do for these types of errors.

Code:
 vmstat
System Configuration: lcpu=5 mem=14336MB

Code:
 oslevel
5.2.0.0

Code:
AAA.sh_20090203.log:/entH/bin/AAA.sh[277]: /usr/bin/rm: 0403-013 There is not enough memory available to run the command.
AAA.sh_20090203.log:/entH/bin/AAA.sh[278]: /usr/bin/rm: 0403-013 There is not enough memory available to run the command.
AAA.sh_20090203.log:/entH/bin/AAA.sh[279]: /usr/bin/rm: 0403-013 There is not enough memory available to run the command.
AAA.sh_20090203.log:/entH/bin/AAA.sh[287]: /usr/bin/rm: 0403-013 There is not enough memory available to run the command.

XXXXX.sh_20090203.log:/entH/bin/XXXXX.sh[106]: /usr/bin/grep: too big
XXXXX.sh_20090203.log:/entH/bin/XXXXX.sh[124]: /usr/bin/grep: too big
XXXXX.sh_20090203.log:/entH/bin/XXXXX.sh[124]: /usr/bin/wc: too big
XXXXX.sh_20090203.log:/entH/bin/XXXXX.sh[129]: /usr/bin/ls: too big
XXXXX.sh_20090203.log:/entH/bin/XXXXX.sh[130]: /usr/bin/wc: too big

YYYYY.sh_20090203.log:/entH/bin/YYYYY.sh[107]: /usr/bin/grep: too big
YYYYY.sh_20090203.log:/entH/bin/YYYYY.sh[125]: /usr/bin/wc: too big
YYYYY.sh_20090203.log:/entH/bin/YYYYY.sh[125]: /usr/bin/grep: too big
YYYYY.sh_20090203.log:/entH/bin/YYYYY.sh[130]: /usr/bin/ls: too big
YYYYY.sh_20090203.log:/entH/bin/YYYYY.sh[131]: /usr/bin/cat: too big

ZZZZZ.sh_20090203.log:/entH/bin/ZZZZZ.sh[107]: /usr/bin/grep: too big
ZZZZZ.sh_20090203.log:/entH/bin/ZZZZZ.sh[125]: /usr/bin/wc: too big
ZZZZZ.sh_20090203.log:/entH/bin/ZZZZZ.sh[125]: /usr/bin/grep: too big
ZZZZZ.sh_20090203.log:/entH/bin/ZZZZZ.sh[130]: /usr/bin/ls: too big

Please advise.
Thanks
Jerardfjay
# 2  
Old 02-05-2009
The problem is not the system memory. The limitation is the memory available to read the large files by the commands you are running (like grep, for example).

These must be very large files.

Can you post the size of the files?
# 3  
Old 02-05-2009
Quote:
Originally Posted by Neo
The problem is not the system memory. The limitation is the memory available to read the large files by the commands you are running (like grep, for example).

These must be very large files.

Can you post the size of the files?
For instance I am posting the typical size of files that got the error on the "rm" command. These are not big at by any means.

Code:
-rw-rw-rw-   1 userid   groupid          266 Feb 05 12:12 AAA.sh_158098.tmp
-rw-rw-rw-   1 userid   groupid          327 Feb 05 12:12 AAA.sh_158098.sql.0
-rw-rw-rw-   1 userid   groupid            6 Feb 05 12:12 AAA.sh_158098.0.AK5File
-rw-rw-rw-   1 userid   groupid           18 Feb 05 12:12 AAA.sh_158098.0.AK2File
-rw-rw-rw-   1 userid   groupid           78 Feb 05 12:12 AAA.sh_158098.0

# 4  
Old 02-05-2009
You are right. That's odd. I'll think about it and post back.

In the meantime, if anyone has a suggestion, please post.
# 5  
Old 02-05-2009
Data not enough memory and too big errors

Neo,

It is very strange. The frequency of the memory errors are more when compared to the "too big" error. One of the items that IBM support asked us to increase for the "too big" error was the value of ncargs from 512 to 1024. (Initially this value was 6). However this did not have a positive impact.

Here is the individual limits for the userid under which the scripts execute.

Code:
 ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        4194304
memory(kbytes)       unlimited
coredump(blocks)     unlimited
nofiles(descriptors) 8000

If anyone has any ideas we are willing to give it a shot. Thx.
# 6  
Old 02-06-2009
Hmm, when a command as small as "/usr/bin/rm" does not fit into memory any more (as per your first post here) there is something seriously wrong.

If the problem goes away temporarily with a reboot chances are you have a memory leak problem. To track this sort of problems down is sometimes hard, because of the ephemeral nature of these. You could wait until the problem shows up and then use "ps -Alo vsz<,other options>" to get the "virtual memory footprint" of every process running. See the man page of "ps" for more information about possible options.

Another common source of problems are java processes, which are generally known to be memory hogs. Find out if there are java processes running ("ps -fe | grep java") and if there are some you have a likely cause for your problem.

Further lets examine the overall memory situation of your machine. Please post the output of the following commands:

"svmon -G" (only as root)

"vmstat -v"

"lsps -a"

Also examine the crontabs of all users on the machine. Maybe some memory hog is started regularly and this is whats causing your problems.

I hope this helps.

bakunin
# 7  
Old 02-09-2009
Question

Quote:
Originally Posted by bakunin
Hmm, when a command as small as "/usr/bin/rm" does not fit into memory any more (as per your first post here) there is something seriously wrong.

bakunin
Bakunin,

I concur too. I am not sure what is causing these errors and they seem to last for a couple of minutes and then they would not appear for the rest of the day. Then they would not occur for a couple of days and then they would prop up. Seems very strange. We do have java processes that are event driven by MQ. In fact our MQ itself has a java wrapper around it. Since I am not a root user I will post everything except svmon -G output.
Here are the rest of the outputs that you

Code:
ps -ef | grep java
     xxx 140596 140870   0 09:12:51      -  0:01 /usr/java131/jre/sh/java -mx256m -ms64m abcde.xx.yy.gpc.gco.cm.mqft.dispatcher.MqftDispatcherProgram /appl01/GPC/conf/mqft.disp.cfg
 userid 156590 135086   0 09:13:01  pts/5  0:00 grep java

Code:
 vmstat -v
              3670016 memory pages
              3570524 lruable pages
               252061 free pages
                    1 memory pools
               222137 pinned pages
                 80.1 maxpin percentage
                  3.0 minperm percentage
                 80.0 maxperm percentage
                 79.9 numperm percentage
              2856298 file pages
                  0.0 compressed percentage
                    0 compressed pages
                 79.9 numclient percentage
                 80.0 maxclient percentage
              2856298 client pages
                    0 remote pageouts scheduled
                10520 pending disk I/Os blocked with no pbuf
                    0 paging space I/Os blocked with no psbuf
                 3026 filesystem I/Os blocked with no fsbuf
                    0 client filesystem I/Os blocked with no fsbuf
                39249 external pager filesystem I/Os blocked with no fsbuf

Code:
 lsps -a
Page Space      Physical Volume   Volume Group    Size %Used Active  Auto  Type
hd6             hdisk45           rootvg        6144MB     1     yes   yes    lv

As soon as I can get one of the UNIX admins to work with me on the svmon -G command I will post the results of that command as well. Thank you for your comments.

Jerardfjay
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. AIX

Apache 2.4 directory cannot display "Last modified" "Size" "Description"

Hi 2 all, i have had AIX 7.2 :/# /usr/IBMAHS/bin/apachectl -v Server version: Apache/2.4.12 (Unix) Server built: May 25 2015 04:58:27 :/#:/# /usr/IBMAHS/bin/apachectl -M Loaded Modules: core_module (static) so_module (static) http_module (static) mpm_worker_module (static) ... (3 Replies)
Discussion started by: penchev
3 Replies

2. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

3. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

4. Shell Programming and Scripting

how to use "cut" or "awk" or "sed" to remove a string

logs: "/home/abc/public_html/index.php" "/home/abc/public_html/index.php" "/home/xyz/public_html/index.php" "/home/xyz/public_html/index.php" "/home/xyz/public_html/index.php" how to use "cut" or "awk" or "sed" to get the following result: abc abc xyz xyz xyz (8 Replies)
Discussion started by: timmywong
8 Replies

5. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

6. Shell Programming and Scripting

cat $como_file | awk /^~/'{print $1","$2","$3","$4}' | sed -e 's/~//g'

hi All, cat file_name | awk /^~/'{print $1","$2","$3","$4}' | sed -e 's/~//g' Can this be done by using sed or awk alone (4 Replies)
Discussion started by: harshakusam
4 Replies

7. Linux

Help req for...shell script to catch "db2 connect" errors ...

Hello friends, Assume that, I am trying to execute a "db2 connect" command from Linux shell prompt via a shell script called "sample" sample db2 connect to bas39 $sample If the database is not present its should display a custom error message by catching the error message given by db2.... (1 Reply)
Discussion started by: frozensmilz
1 Replies

8. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies

9. UNIX for Dummies Questions & Answers

No utpmx entry: you must exec "login" from lowest level "shell"

Hi I have installed solaris 10 on an intel machine. Logged in as root. In CDE, i open terminal session, type login alex (normal user account) and password and i get this message No utpmx entry: you must exec "login" from lowest level "shell" :confused: What i want is: open various... (0 Replies)
Discussion started by: peterpan
0 Replies
Login or Register to Ask a Question