AIX

Hello Gurus,
I am using AIX 5 and on running topas command. I can see the computational memory is 93.3% with Swap Paging memory at 2.2%. Could you please advise if there is any impact by the growth of computational memory?

Below is the stat:

Code:

MEMORY
Real,MB   22528
% Comp     93.3
% Noncomp   6.6
% Client    6.6

PAGING SPACE
Size,MB   17376
% Used      2.2
% Free     98.8

Thank you

AIX (and most other Unix-systems, for that matter) try to put as much of the available memory to good use as is possible. That means: if a system doesn't need memory for programs it will increase the size of all sorts of buffers, caches, etc. to improve performance. Once more memory is demanded by running programs these buffers will decrease in size automatically.

So in short: the high memory consumption is absolutely normal and is to be expected. The only possible reason for concern is when the output of "vmstat" shows non-zero values in the "pi" and "po" (page in / page out) columns,

Search this forum for "performance" and "vmstat" to find lots of threads where this issue (and related issues) is discussed in much more detail.

I hope this helps.

bakunin

well let me disagree here ... computational memory above 85% is never a good idea - filecaching is important for almost every workload - every IO needs to be cached and this is done with non-comp memory - and if you have for example an oracle DB with 90% comp memory, you can kiss performance goodbye as the system is scanning/freeing itself to death. I would add memory to get at least below 80%

regards
zxmaus

a) Could there be a memory leakage in application leading computational memory increasing so fast?
b) Could it lead to system crash if computational memory reaches what percentage?

Thank you

We do not know which AIX version you are running, we do not know your vmo settings, we do not know what is running on your server, so its a little hard to tell what may or may not cause your computational memory utilization and if it is a leak or normal growth. Since AIX has a dynamic kernel, the memory utilization grows over time which is normal, and many workloads are preallocating memory - Websphere, Oracle, Sybase, DB2, SAS just to name a few.

Depending on your AIX version and tunable settings you may start paging out computational content - which usually slows down your corresponding application as AIX is paging in that case entire processes to disk - it stops the process, moves it - and than continues using it.

When you are that high in comp memory, that means you have literally no room in memory for non-comp (file caching) what is as well vital for many applications. If they are requesting memory, than the system starts thrashing (excessive scanning / freeing memory) what may cause a huge amount of cpu overhead and slowdown of your box - this can go that far that no other processing happens on the corresponding server.

Since you are already starting to page (2% paging space used) you are at risk that when you filling your paging space, your system might at first randomly kill processes to survive and later either hang with no login possible or crash and restart itself to free up memory.

What you need to do is monitor your system very closely - like if the amount of comp memory in use is growing and if your paging space utilization is growing.

IBM recommends to have comp memory around 75% (and from a performance perspective I can only confirm this - my boxes - DB servers - perform best when they are somewhere between 66 and 80%) - depnding on the criticality of your applications, buying a few gig of memory may be cheaper than risking a crash.

Hope that helps
kind regards
zxmaus

Thank you very much zxmaus.

I am running AIX 5 and it has applications talking like tuxedo, websphere, oracle 11g.

a) Can you give me the URL where IBM recommends to have comp memory around 75%?
b) At what percentage of computational memory - Applications running on it or Server will crash? Or computational memory and paging space both can keep growing and things will hang/crash only when paging memory is around 90%. Basically trying to understand the importance of computational memory.

Thanks again!

Hi,

in about 2000 PMRs which I raised in the past 15 years for performance issues But I will see if I find something on the pages.

I think you need to understand what computational memory is - and what computational memory NOT is to understand why it is important to have sufficient non-computational memory in your system?

Computational memory = basically everything that does not exist on your hdisk - for example DB content changed but not yet committed, anything volatile that is work in progress - basically everything that will be gone when your system crashes.

Non computational memory is everything that has a place on disk but still needs to be working in memory - like shared libraries, executables, ... on top of this each and every disk IO needs 1 page of non-computational memory - so every read and every write.

Usually your system is hopefully tuned in a way that computational memory is NOT paged to disk before you reach 97% (minperm=3%). As well your lrud_file_repage is hopefully set to 0. That means until you do not exceed 97% computational memory, your system will page when it needs to fulfill IO or filecaching needs, or if you start a process or you carry out a job (batches, backups and much more) - but it will not page out for example the DB processes. Nonetheless it will fill up your pagingspace.
If your computational memory needs exceed 97% than the system does not care anymore if what is in memory and needs to be paged out is your database- or application data or if its filecaching - it will just page out everything.
You are saying you are running oracle. Each oracle process doing work can consume 1 pp worth of (shared) memory. In our company that is about 64 mb as we have rather small disks - but this can be 128, 256 or more MB in certain setups - and oracle can have quite a few processes (some of our DB servers have a few thousand). Oracle is a 64bit application, so shared memory is only limited by the amount of physical memory available in the box - so this should not cause application issues - BUT Oracle has on top of its processes as well its SGA - a fixed amount of memory that is allocated for oracle only - so neither the kernel nor any other application running on your system will have any access to it. And - if the SGA is not locked, it may potentially be paged out in total to pagingspace - if this happens, it is kind of disatrous for your DB performance.
On top of that you are running Tuxedo (weblogic) and websphere on your server. I do not know for sure but I believe that Tuxedo is a 32 bit application - I do know for sure that Websphere is. And both are preallocating predefined additional shared memory for themselves. Problem with 32bit applications is, that they cannot use unlimited shared memory - AIX offers only 13 shared memory segments for their use (this can be amended with certain system settings to be exported in the environment but I would not assume that has been done in first instance). And ... 32bit applications use different shared libraries than 64bit applications, so the system needs to hold even more content of this kind in (non-computational) memory - which makes your non-comp needs even bigger. Generally its not such a great idea to mix DBs and Webapplications on the same server for various reasons - the above is one of many, the different needs in workload (DBs = large parallel serial threads, Webapplications lots of mini-threads and random with corresponding loads of IOs ) ...

If you like, you can post a vmstat -Iwt 2 10 output taken at a busy timeframe and vmstat -s output and maybe svmon -G here ... that will help me explain a little further ?

Interesting to see would be as well the top area of commands like svmon -U oracle and similar for your other application users assuming that Tuxedo and Websphere are running under their own ID ? You will be surprised how much that adds up.

You maybe want to have a look here as well.

Kind regards
zxmaus