Solaris

Can you help. My server sunning solaris 9 on x86 platform pretty much hung for a few hours... I could not use telnet or ssh to the box - it kept refusing connection. A few hours later - I was able to log in again.

The server has not rebooted but here are the first errors in the messages log file:

Mar 17 19:01:01 host11 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Mar 17 19:06:13 host11 last message repeated 154 times
Mar 17 19:07:39 host11 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Mar 17 19:11:00 host11 last message repeated 606 times
Mar 17 19:11:00 host11 genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 27435 (ssh)
Mar 17 19:11:12 host11 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Mar 17 19:15:01 host11 last message repeated 3 times
Mar 17 19:16:00 host11 genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 15140 (cron)


As far as I can see - everything looks fine currently.

# swap -s
total: 54104k bytes allocated + 63904k reserved = 118008k used, 1308268k available
# df -k /tmp
Filesystem kbytes used avail capacity Mounted on
swap 1308716 124 1308592 1% /tmp

Is there anything I can to find out what caused this error?
And is there anything I can do to prevent it occuring again?

I have not rebooted the server in 7 days - so hopefully there might be some evidence somewhere as to what caused it. Any help is appreciated.

I would look for cron jobs or at jobs that started at 19:00.

we have a few cron jobs that run every 5 - 10 every hour of every day.
They have been running for over 6 months now without any issues...
Not doing anything fancy, all using standard unix commands, mainly pinging other nodes or ftping some files etc...


Is there anything else I can check?

Also - can /tmp or swap be configured differently to help avoid this ?

You don't need to be fancy or use non-standard commands to have a problem. Scripts can run fine for a long time and then encounter data they can't handle well. As one example, if you ftp a file to /tmp and the file is unexpectedly large it would result in this exact problem.

ok, fair point.

Can I change it so that filling /tmp doesnt kill swap and therefore my server?

You can add a size=500m option or whatever to the /tmp entry in vfstab. Could be the other way around though. Maybe VM filled swap rather than files in /tmp.