UNIX for Advanced & Expert Users

I have a Sun V880/ Solaris 8/ 16 G of memory

We previously had 8 G of mem. Started getting fork errors. Thought problem was becauseit was low on memory. Upgrade to 16 G of mem. Still, box is getting fork errors. Not seem to be using over 7 G of memory. We think it's the PeopleSoft and Oracle application problem related.

Is there some tool or software that uses a lot of memory? Like over 2 G? I want to use this memory to see if the box is allowing me to go over this 7 G amount.

When i do a top:

Memory: 16G real , 8777 free, 7000m swap in use , 35 G swap free.

I need some sort of application to test that uses a lot of memory. This will determine if the box is allowing me to use all these available memory.

I have messed with the /etc/system parameters settings and it is not the problem

Any input on this problem would be greatly appreciated.

Thanks

Let's take another look at the system file just to be sure...

I'm guessing that you have maxusers set way too low. Here is a command to display the current value, but you must type this correctly because it is a dangerous command:
echo "maxusers/D" | adb -k /dev/ksyms /dev/mem

This is controlled by a setting in /etc/system, but you should probably not set it at all. It should default to 2048 on your system.

One of the values controlled by maxusers is max_nprocs. A maxusers of 2048 will result in a max_nprocs of 30000. You are probably bumping in to max_nprocs. You can display that with:
echo "max_nprocs/D" | adb -k /dev/ksyms /dev/mem

max_nprocs is the maximum number of processes allowed on the system. Once that limit is reached, fork() will fail.

Thanks for your reply Perderabo.

After running the command:

echo "maxusers/D" | adb -k /dev/ksyms /dev/mem

I get:

maxusers:
maxusers: 2048


When I run:

echo "max_nprocs/D" | adb -k /dev/ksyms /dev/mem

I get:

max_nprocs:
max_nprocs; 30000


maxusers or max_nprocs is not specified in my /etc/system parameters. So therefore it is set as default.

So, you are right. I am bumping into the max number of allowable on the system.

Is there anyway for me to increase this value? Maybe to double the value?

Thanks again Perderabo.

when i do:

ps -ef | wc -l

I get:

300

Which is not 30000.

Dang! I thought that this was going to be an easy one....

First of all , maxusers=2048, and max_nprocs = 30000 are the correct values for your system. There is one more process slot oriented parameter and:
echo "maxuprc/D" | adb -k /dev/ksyms /dev/mem
will display it. It should be 29995 for you. Please verify that.

I assume that you are running a 64 bit kernel, but let's make sure:
isainfo -kv

The best command to look at swap is:
swap -s
what does it say?

Are you actually maxing out at 300 processes????

What message do you get when the problem occurs? How do you fix it? Do you have a system console? Any messages on it? Can root still run some commands?

If you can, very soon after a forking problem run:
vmstat 1 5
kstat -p "cpu_stat:0:cpu_stat0:procovf"
echo "fork_fail_pending/D" | adb -k /dev/ksyms /dev/mem
and post the results.

Also, post the output from:
grep swap /etc/vfstab
df -k /tmp
swap -l

# echo "maxuprc/D" | adb -k /dev/ksyms /dev/mem
physmem 1f3415
maxuprc:
maxuprc: 29995

# isainfo -kv
64-bit sparcv9 kernel modules

# swap -s
total: 4384192k bytes allocated + 1722416k reserved = 6106608k used, 37170768k available

No, I was wrong, I am not maxing out at 300 processes.

The message occurs when I try to start another PeopleSoft App. I currently have 5 PeopleSoft Application running. I get the fork errors when I try to start the 6th one. So, to get around it, I shut off one of the PeopleSoft App to start another one. I want to start all 6. Soon, I will need to add 5 more Apps. I ran 5 oracle database with 10 Peoplesoft App one time on a 4G mem system. There was no problem. I did not get this fork error with the 4G system. Therefore, I think it has something to do with the system, not letting me using more than 8G. Maybe some sort of parameters or something is wrong. When you upgrade your system memory, all you do is manually add the physical memory into the box right? No need to configure anything special right? I believe the box will automatically detect the memory and let you use it, correct? But, even before we upgrade to 16G, the box never really used all of the previous 8G of memory. It always get to around 7 to 7 ½ G.

There is no message on the system console. I only get the message when I try to start my PeopleSoft processes. The system does not freeze or anything. Just fork error and system continues to run. Does not matter what user you are logged in as.


I ran these right after the fork error:

Code:

psoft:fshw > vmstat 1 5
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 m3   in   sy   cs us sy id
 0 0 0 37946976 11757256 112 422 311 4 1 0 0 0 0  1  0 1102 4396 1193  6  2 92
 0 0 0 37073088 10833640 14 510 0 0 0 0  0  0  0  0  0 1782 37611 1839 18 3 79
 0 1 0 37069600 10830040 42 670 0 0 0 0  0  0  0  0  0 5293 8597 2602 32  7 61
 1 1 0 37069280 10829704 114 653 0 0 0 0 0  0  0  0  0 5950 18070 6655 31 7 62
 0 0 0 37069480 10829848 0 0 0  0  0  0  0  0  0  0  0 1839 2007 1246 18  6 75

psoft:fshw > kstat -p "cpu_stat:0:cpu_stat0:procovf"
cpu_stat:0:cpu_stat0:procovf    0

psoft:fshw > echo "fork_fail_pending/D" | adb -k /dev/ksyms /dev/mem
Cannot adb -k: cannot open /dev/kmem


psoft:fshw > grep swap /etc/vfstab
/dev/md/dsk/d5  -       -       swap    -       no      -
/dev/dsk/c1t1d0s1       -       -       swap    -       no      -
/dev/dsk/c1t2d0s1       -       -       swap    -       no      -
/dev/dsk/c1t4d0s1       -       -       swap    -       no      -
#/dev/dsk/c1t5d0s1      -       -       swap    -       no      -
swap    -       /tmp    tmpfs   -       yes     -

psoft:fshw > df -k /tmp
Filesystem            kbytes    used   avail capacity  Mounted on
swap                 37067824      72 37067752     1%    /tmp

psoft:fshw > swap -l
swapfile             dev  swaplo blocks   free
/dev/md/dsk/d5      85,5      16 8395184 8395184
/dev/dsk/c1t1d0s1   118,17     16 16780208 16780208
/dev/dsk/c1t2d0s1   118,1      16 16780208 16780208
/dev/dsk/c1t4d0s1   118,41     16 16780208 16780208

I don't think the problem is swap related.


Here is the error message:

Booting admin processes ...

exec BBL -A :
Failed.

tmboot: CMDTUX_CAT:827: ERROR: Fatal error encountered; initiating user error handler

exec tmshutdown -y

Shutting down all admin and server processes in /opt/psoft/ftst/appserv/fshw/PSTUXCFG

tmshutdown: internal error: CMDTUX_CAT:764: ERROR: can't attach to BB


tmboot: CMDTUX_CAT:828: ERROR: Cannot create error process - fork() failed

One thing that I can try is to use some sort of application that uses a lot of memory and test my machine memory. Do you know of any sample/test application that does this?



Thanks