As promised, here is the second part of the answer:

In the following, when not stated explicitly otherwise, the unit we measure in are memory pages, which is 4K (4096 bytes) in the overwhelming majority of cases. There is the option of using "large pages" (16MB) but i have yet to these used in reality.

I said above that memory is used for programs and what isn't used for programs is used for buffers. IBM calls that computational memory (the part which is given to running processes) and file memory (the part used for file caching). It is obvious that with the constant loading and unloading of processes the memory is given back and forth as the current amount of used memory changes. Also notice that there is "pinned memory". That is memory which is allocated in a way that it can't ever be swapped out. I gave you already one example - shared memory - but there are other types of pinned memory too.

So, at a given moment, save for this pinned memory, we have a memory that is to some part computational memory and (nearly) all the rest is given to file memory. The not pinned memory is constantly scanned for possible candidates to either swap out or to add to either file or computational memory. There is a special process for this, the "lrud" (least recently used daemon). Pages which could be used for file or computational memory (basically the memory that could be swapped) are called "lruable" (notice that all these words are IBM speak, you will find these terms in the output of various programs).

Now suppose we have a system that is running at capacity limit (all other cases are rather trivial): the kernel now has to decide at any given point, what to do: either shrink the file memory or swap some programs out. At first, as ever more memory is demanded, it makes sense to shrink the file cache. At a certain point read/write operations would suffer so much from a further reduction of this cache that it might be overall better to swap something out and back in later. The procedure is called "page stealing" because memory pages which are in use are "stolen" and put to another use. What the kernel does under which circumstance is controlled by the tuning parameters minperm and maxperm. Both these values are percentages.

When the percentage of memory holding file pages falls below minperm, needed memory will come from both file and computational memory. Processes might get swapped out (stealing from computational memory) to avoid reducing the file memory any more. Again, which specific pages are stolen will be determined by the lrud.

When the percentage of memory holding file pages is between minperm and maxperm only file pages get stolen. Processes that are starting get their memory by the kernel diminishing the file memory and giving the so freed memory to them.

This way you can control the decisions a kernel will take under certain circumstances.

You haven't posted any vmstat-output yet, so i take one from the IBM site:

Code:

# vmstat -v
              1048576 memory pages
               936784 lruable pages
               683159 free pages
                    1 memory pools
               267588 pinned pages
                 90.0 maxpin percentage
                  3.0 minperm percentage
                 90.0 maxperm percentage
                  5.6 numperm percentage
                52533 file pages
                  0.0 compressed percentage
                    0 compressed pages
                  5.6 numclient percentage
                 90.0 maxclient percentage
                52533 client pages
                    0 remote pageouts scheduled
                    0 pending disk I/Os blocked with no pbuf
                    0 paging space I/Os blocked with no psbuf
                 2228 filesystem I/Os blocked with no fsbuf
                   31 client filesystem I/Os blocked with no fsbuf
                    0 external pager filesystem I/Os blocked with no fsbuf
                 29.8 percentage of memory used for computational pages

First things first:

Code:

              1048576 memory pages
               936784 lruable pages

We have ~1Mio memory pages, which is 4096*1Mio=4G of real memory. Of these about 936k pages are eligible to be scanned by the lru-daemon.

Code:

                 3.0 minperm percentage
                 90.0 maxperm percentage
                  5.6 numperm percentage
                52533 file pages

maxperm is at 90%, minperm is at 3%. (This is a setting typical for databases, which do not need the file cache for their operations and in fact bypass it.) Right now 52533 pages (~ 200MB) are used for the file cache, which is ~5.6% ("numperm") of the memory: Also 29.8% of the memory is used by programs ("percentage used for computational pages"). I suppose this machine was started only recently because otherwise the file cache would be bigger. Right now the system doesn't know what to put into cache that makes sense.

Another value that is worth checking is seen when issuing vmstat -vs: "Revolutions of the clock hand". This value shows the number of times the lrud has scanned the whole memory completely for potentially stealable/swappable page candidates. The value itself increases over time and is not interesting as such but interesting is the growth rate of the value. The faster it grows the faster the memory is scanned - to put it in dramatic terms: the more frantic the system is searching for freeable memory. If the system searches frantically there must be some pinch it is in, yes? This way you can anticipate the development of a memory shortage even before the system starts really swapping. Of course the numperm and number of computational pages add to this picture but it is always a good idea to take pictures from different sides and look if all adds up from every side.

OK, here i end. There is a lot more to say about each of these things but, again, if you are really interested, you should take a class or two and you might also search our forum for the terms you now should be familiar with. Several people have written valuable info about these things over time and you can learn from others perhaps more than from me. If you still have specific questions you can also ask, we are glad to help - just don't expect every answer to be as long as this one.

You might want to read my performance tuning introduction for more general information about the concepts and tools involved in performance monitoring and tuning.

I hope this helps.

bakunin