Solaris

This is completely normal.

Cache memory is free memory with benefits. Having nearly 100% memory as cache means you have nearly 100% memory free.

If you're dropping caches all the time in linux you're throwing away your disk cache all the time, forcing your disk to be read again to get them back into memory when needed.

@Corona688:
... provided the Linux VM would be perfect.
Do continues I/O until the buffer cache is near 100%, and while doing further I/O start a big Java application. It will take AGES, even an OOM can occur. At least on some Linux kernels.
Do the drop_caches trick, and the Java application will start FAST.

THANK YOU FOR YOUR ASSISTANCE
Rgds.
Terry

Sounds like time to update the OS to one where VM is more robust.

There are tricks to wire pages, so you want to discourage your developers from doing this -- it's mostly just hoarding.

I'm a little late to this thread, but what kind of file system(s) are being used on the servers?

Oracle DB processes tend to request large memory pages, while the ZFS ARC uses small memory pages.

When memory gets tight - such as the servers in question - the kernel spends an inordinate amount of time coalescing large pages for Oracle then breaking them apart for ZFS.

Over and over and over.

Try running this dtrace script when things are slow:

Code:

#!/usr/sbin/dtrace -s

#pragma D option quiet

profile:::profile-1001hz
/ arg0 /
{
    @hot[ arg0 ] = count();
}

dtrace:::END
{
    printa( "%@u %a\n", @hot );
}

That will tell you where your kernel is spending it's time. If you see lots of "coalesce" in the output after you break a running copy of that script, that's your problem.

Free up some memory - lower your Oracle SGA, severely limit your ZFS ARC size, etc.

ZFS Evil Tuning Guide - Siwiki seems apropos.

Sounds like it's time to upgrade your kernel, then.

And this has nothing to do with it refusing to drop cache. It's because java is not in cache; other activity has gradually cleaned it out, and everything must be reloaded from disk, while the disk is already busy. Relying on these tricks is a bad idea. Something more fundamental is out of whack.