AIX

Hi,

please see this - I have added a few explanations ...

kthr: Information about kernel thread states.

r: Average number of runnable kernel threads over the sampling interval. Runnable threads consist of the threads that are ready but still waiting to run, and the threads that are already running. Number should not be higher than the number of lcpus in your system.

b: Average number of kernel threads placed in the Virtual Memory Manager (VMM) wait queue (awaiting resource, awaiting input/output) over the sampling interval. Any value here points to insufficient filesystem buffers, problems with IO subsystems or overall insufficient memory.

p: Number of threads waiting on I/O to raw devices per second. Only valid if you have rawdevices. Any number here points to problems with IO subsystem.

Memory: Information about the usage of virtual and real memory. Virtual pages are considered active if they have been accessed. A page is 4096 bytes.

avm: Active virtual pages. So called computational memory. In our environment, we have best performance if avm is around 70% on oracle systems, 80% on sybase systems. Ideally it does not exceed "100% - numperm%" after your system is sufficiently tuned.

fre: Size of the free list. The amount of really free memory in the box. That is - memory that is not used either computational or for filesystem caching. This number should be sufficiently high to accomodate the requested memory at any given time.

Note:
A large portion of real memory is utilized as a cache for file system data. It is not unusual for the size of the free list to remain small but it is vital that the free list NEVER drops to 0.

Page: Information about page faults and paging activity. These are averaged over the interval and given in units per second.

fi: File page-ins per second. These are e.g. all your disk reads.

fo: File page-outs per second. These are e.g. all your disk writes.

pi: Pages paged in from paging space. BAD as disk space is obviously slower than memory. Usually points to bad system tuning or insufficient memory.

po: Pages paged out to paging space. BAD as disk space is obviously slower than memory. Usually points to bad system tuning or insufficient memory.

fr: Pages freed up by page replacement and made available in the free list.

sr: Pages scanned by page-replacement algorithm to determine if they can be freed up.

Note: it is not important if these numbers are high or low. Important is the ratio of sr:fr - you have better performance the lower this number is - ideally not exceeding 1:2 - if you are higher than 1:8 you usually are in big trouble.

cy: Clock cycles by page-replacement algorithm. VMM uses a clock-algorithm to implement a least recently used (lru) page replacement scheme. Pages are aged by being examined by the clock.

Faults: Trap and interrupt rate averages per second over the sampling interval.

in: Device interrupts. Usually IO counter. This number usually presents the number of pages which will be adressed in the next cycle by the free list. If the free list is too small to adress these needs, memory needs to be scanned and freed for the next IO to occur.

sy: System calls. The 'work' your system is doing.

cs: Kernel thread context switches. Number of times your kernel is starting to do something else.

CPU: Breakdown of percentage usage of processor time.

us: User time. Amount of real work done by your cpu for applications - like your DB

sy: System time. So called kernel cpu usage. High values point to a lot of overhead of some kind. If you are memory constrained, cpu cycles will be used to free up memory. On sybase systems high values may represent cpu spinning - check if you have the correct number of engines for your DB.

id: Processor idle time. CPU cycles where your system is doing really nothing.

wa: Processor idle time during which the system had outstanding I/O requests. Usually a bad sign. Check if you can implement async IO which allows your cpu to process without waiting for IO to finish first. Might point to problems with your IO subsystem, insufficient buffer caches.

Your system - even though the AVM value is still in a rather acceptable area - seems as it would benefit from more memory - especially for filecaching and your free list. You should make sure that your system stops paging and you should try to reduce the amount of wait cpu.

If your box would be mine, I would implement below values and see if things improve. I would add as well at least 1-2 GB memory.

Code:

vmo -p -o minperm%=5
vmo -p -o maxperm%=90
vmo -p -o maxclient%=90
vmo -p -o minfree=1000
vmo -p -o maxfree=1200
vmo -p -o lru_file_repage=0
vmo -p -o lru_poll_interval=10
ioo -p -o hd_pbuf_cnt=1024
ioo -p -o numfsbufs=1024 ### can go up to 2048 if needed

I hope this helps,
kind regards
zxmaus

I am not sure if I am up to date (stuck in AIX 5.3 sadly), but I thought that hd_pbuf_cnt is kind of obsolete to the use of pv_min_pbuf.
Also numfsbufs I always thought is for JFS and for JFS2 you use j2_dynamicBufferPreallocation. If this is not sufficient it is recommended to also tune j2_nBufferPerPagerDevice in a second step.

You probably shouldn't have asked Google, but unix.com (*snicker*):

AIX 5.2 performance question
"too big" and "not enough memory" errors in shell script
Should I be worried about my AIX Cluster?
How do I boost the Linux performace

...and probably a gazillion of even more informative threads.

I hope this helps.

bakunin

PS: On a personal note I'd like to say for the record that i really appreciate zxmaus to be with us again. I hope everything is well and you have a good time.

bakunin,

thank you - glad to be back

zaxxon

hd_pbuf_cnt=1024 = AIX 5.2 ,
pv_min_pbuf=1024 = AIX 5.3 and later

regarding the buffer tunables, this box seem to have enough jfs2 buffers but not enough buffers for its jfs filesystems, that is why I would suggest to only tune these ?

I am with you setting the other values would not harm the box - I just cannot see at the moment that it is needed. It might be later after initial tuning.

Just for completeness, if this would be my box I would most likely go with

Code:

ioo -p -o j2_maxPageReadAhead=128
ioo -p -o maxpgahead=16
ioo -p -o j2_maxRandomWrite=32
ioo -p -o maxrandwrt=32

on top of the memory tunables from the previous thread - but these are settings you should be rather careful with as they can potentially do more harm than good.

Kind regards
zxmaus

Ah he also wrote it
Never mind

---------- Post updated at 07:09 AM ---------- Previous update was at 06:39 AM ----------

I checked last days all of my documents I gathered and notes I wrote up and tried to get a clue.
I found this table (Table 2.) some minutes ago which sums it up quite well and very clearly - maybe there is someone else who is/was confused like me (ok, it's just me )

Optimizing AIX 5L performance: Tuning disk performance, Part 3

This is another SYSTEM, it is IBM P 550 with 6 Processors and with Memory 96GB / 32GB Paging Space. 1000 users connecting concurrently to the Oracle Database

This is one of the best topics and most useful one as many companies buy AIX machines solely for Oracle Database.


The machine is still running slow. How can one trace and fine tune ?

Code:

 Disk    Busy%     KBPS     TPS KB-Read KB-Writ  Steals     3458  % Comp     90.0
  hdisk1   98.5     1.6K  333.5     1.2K  446.0   PgspIn      307  % Noncomp   9.9
  hdisk2   88.5    10.8K  757.5     3.1K    7.7K  PgspOut     111  % Client    9.9
  hdisk3    5.5   412.0    16.0     4.0   408.0   PageIn     1095
  hdisk4    0.0     0.0     0.0     0.0     0.0   PageOut    2179  PAGING SPACE
                                                  Sios       3271  Size,MB   33408
  Name            PID  CPU%  PgSp Owner                            % Used     35.5
  oracle      3141878  14.2  19.4 oraprod         NFS (calls/sec)  % Free     65.5
  oracle      3313978   5.9  34.9 oraprod         ServerV2       0
   
   
   
   
   
  Paging space is 35 .5 % 
   
  Hdisk 1 (which is internal server hard disk  not storage )is used with 98 %
   
  Hdisk2 is storage disk (which had DB )

Hi,

can we please have as a starter vmstat -Iwt 2 10 and the usual vmstat -s and vmstat -v outputs ?
In this case I would like to see iostat -Dl too, vmo -a, and I would like to know what is set within oracle - SETALL or async or nothing at all - and output of mount command would be interesting also

The full program ... with no data we cannot recommend anything.

Hdisk1 is so busy because your box is paging like hell - usually that means that either your vmo settings are still wrong, SETALL is not set or you really still have not enough memory

Regards
zxmaus