Red Hat

Friends,

I have 8CPU/24gb RAM in my Linux servers around 30+. All the servers are running with Java based application with NFS mounted.

The problem what I have is, When I reboot servers the server memory looks decent. But after some time the "cached" memory increasing and "free" memory getting reduced and reaching some mega bytes. I searched in various forums that it is behavior for RHEL and there wont be any performance issue. But I see better performance when there is more memory available in "free". The same performance goes down when the "free" memory reach 300mb.

Code:

$ free -g
             total       used       free     shared    buffers     cached
Mem:            23         23          0          0          0          9
-/+ buffers/cache:         13         10
Swap:            1          0          1
$ free -m
             total       used       free     shared    buffers     cached
Mem:         24248      23756        492          0        233       9684
-/+ buffers/cache:      13838      10409
Swap:         2047        214       1833
$

I need your expert advice why the cached memory is increasing.. and how I set threshold so that I will always have 1-2gb free.?

Thank you
Bala

This is indeed normal behavior with the Linux OS, which uses unused memory for caching and buffering. The moment that memory is need for something else, it will be freed immediately.

The important line to look at is the +/- buffers cache line, which shows that around 14GB is in use and 10.5 GB is free..

There normally shouldn't be performance issues, with or without the memory used for caching. How are you measuring this?

When you reach 300 MB, how much caching is in use then?

Thank you for your response.

I completely agree with your statement. But i can clearly notice performance issue when the "free" memory on first line goes down.

Is there a way i can reserve free memory to at least to 2g?

Thanks
Bala

I don't believe that the performance degradation has anything to do with the "free" (which in fact is: unused) memory going down. The reason for it going down is simply because at start time the system has no idea what to put into the cache. Over time it "learns", by loading disk content to memory, which contents are used more often and therefore puts them into cache. This does not slow down operations but in fact speeds them up!

Your problem may come from several reasons: most likely is a "memory sink", which is a pure application problem. The application requests memory dynamically from the system and the system fulfills this request. But the application (being written poorly) doesn't keep track of this memory, so, instead of reusing it or giving it back it requests another chunk of memory from the system - and so on, ad infinitum. Because the systems memory is limited this can't go on forever and over time the systems resources are exhausted. A prominent instance for this happening was/is Firefox. If you run it for several weeks you will notice that it has allocated several GB of memory even if you have only one (empty) window open.

The only rememdy for this (if you can't fire the programmer for sloppy work and then get another one to change the application, which would be the best course of action) is to restart the application (not the system!) at certain intervals. Which these intervals are depends on the application and how rapidly it gobbles up the memory.

I hope this helps.

bakunin

But a "memory sink" would not be "system buffers/cache". It is part of the 13 GB memory in the sample in post#1.
Linux really suffers from the following work-load:
one process keeps I/O writing (until the free memory is shrunk to nearly 0), while another process writes to not yet initialized memory (that has been malloc()ed before).

This is true, but the "memory sink" might over time deplete available memory so much that cache and free mem is reduced to (near) zero thus slowing down operations.

Very interesting, i haven't known that! But even then, retaining any arbitrary amount of memory (as thread-o/p's suggest 2GB) as free will not help anything. It will just ensure that the point at where this problem hits will be reached sooner.

I hope this helps.

bakunin