Linux

I had a situation where the server was basically unresponsive and apparently it was out of memory. I ran top and free commands and got the following outputs. I figured it was out of memory because there was no free swap left. But quite frankly I'm perplexed by this output, why does it appear that there's 18845 MB of free memory? I must be missing something.

Code:

# top

top - 10:40:03 up 13 days, 14 min, 21 users,  load average: 6.95, 9.27, 49.59
Tasks: 20984 total,   5 running, 20979 sleeping,   0 stopped,   0 zombie
Cpu(s): 10.3%us, 15.5%sy,  0.0%ni, 53.9%id, 19.9%wa,  0.0%hi,  0.4%si,  0.0%st
Mem:  24554800k total,  5376404k used, 19178396k free,      932k buffers
Swap:  2097144k total,  2097144k used,        0k free,   121164k cached

[snip]

# free -m
             total       used       free     shared    buffers     cached
Mem:         23979       5258      18720          0          0        123
-/+ buffers/cache:       5133      18845
Swap:         2047       2047          0

After some troubleshooting I found a stale NFS mount and there were thousands of hung command because of it (we ran some scripts every minute to get df usage - don't ask). Apparently they took up all the memory - after I killed them, it freed up half the swap space and the server was responsive again.

But I'm confused why it appeared I had almost 18GB of free memory. If someone could explain how to interpret the above numbers it would be great!

Probably a high swappiness, if this is Linux, read from /proc/sys/vm/swappiness. Low value will favor dropping pages in cache, high value will write more to swap. Since you had lots of stale commands, the kernel figured they did not need to be in active memory.. But since you have so much free, I wonder why it swapped at all?

Forgot to mention, this is RHEL 5.5 (64-bit). 24GB of physical memory.

Yes I agree the server was extremely slow because of constant swapping which I figured by 0 free swap space.

To be honest I have more experience with Solaris than linux, and I've seen similar issue on Solaris (no free swap memory so the server is extremely slow). However, in such situation I would see very low free memory (less than 1 GB). What puzzled me is why I had 18GB free and the server was acting as if it was out of memory. Unless I'm reading the output of these commands wrong?

As a rule of thumb swap should be roughly equal to installed memory and at least twice the size of the largest process. It is no longer necessary to make swap twice memory (as was required with early unix). Linux uses swap continuously to optimise memory by swapping pages. If swap is too low, the Operating System will fail to function efficiently and may even stall.

With a rogue cron it is easy to exceed the kernel maximum number of files open by one user. If it is a root cron you are heading for a crash.