Lots of page faults on AIX mySQL lpar Post: 302477334

Sponsored Content

Operating Systems AIX Lots of page faults on AIX mySQL lpar Post 302477334 by crosys on Saturday 4th of December 2010 11:23:21 AM

12-04-2010

Registered User

Hi,

Thanks on fast replay. My colleague started to complain that system is suddenly become very slow. I did some nmon stats and find out that system is generating more than 13000 I/O in sec and lots of page faults. I did some adjustments on AIX OS;

Code:

vmo -p -o minperm%=3
vmo -p -o maxperm%=90
vmo -p -o maxclient%=90
vmo -p -o minfree=960
vmo -p -o maxfree=1088
vmo -p -o lru_file_repage=0
ioo -p -o pv_min_pbuf=1024

I also edit /etc/filesystems and use 'dio' because AIX lpar is attached to SAN so I want to take advantage of caching, throughput optimization, and performance by enabling file system direct I/O on AIX. This, according to my knowledge, will allow the file system buffer cache to be bypassed which eliminates the double buffering of data with mySQL which can adversely affect file system I/O performance. I tried to upload excel zip file of nmon statistic but error always appeard (file is less then 3 MB). Problem still remain, slow performance.

Some new stats are:

Code:

 # vmstat -s──────────────────────────────────────────────────────────┘
               788510 total address trans. faults
                49725 page ins
                11898 page outs
                    0 paging space page ins
                    0 paging space page outs
                    0 total reclaims
               440936 zero filled pages faults
                 4617 executable filled pages faults
                    0 pages examined by clock
                    0 revolutions of the clock hand
                    0 pages freed by the clock
                60589 backtracks
                    0 free frame waits
                    0 extend XPT waits
                19289 pending I/O waits
                61623 start I/Os
                32640 iodones
               133924 cpu context switches
                35253 device interrupts
                16549 software interrupts
                78787 decrementer interrupts
                 1491 mpc-sent interrupts
                 1491 mpc-received interrupts
                  300 phantom interrupts
                    0 traps
               958258 syscalls

Code:

 # vmstat 5 5─────────────────────────────────────────────────────────┘
System configuration: lcpu=8 mem=5120MB ent=1.80
kthr    memory              page              faults              cpu
----- ----------- ------------------------ ------------ -----------------------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
 1  0 314163 966357   0   0   0   0    0   0   2  210 170  0  0 99  0  0.01   0.3
 1  0 314163 966357   0   0   0   0    0   0   0   30 159  0  0 99  0  0.01   0.3
 1  0 314172 966348   0   0   0   0    0   0   1  364 165  1  1 98  0  0.04   2.3
 1  0 314172 966348   0   0   0   0    0   0  47  138 262  0  0 99  0  0.01   0.5
 1  1 314228 966291   0   0   0   0    0   0 311 1517 846  1  1 97  1  0.05   2.5

Code:

Topas Monitor for host:    mysql01              EVENTS/QUEUES    FILE/TTY
Sat Dec  4 17:15:49 2010   Interval:  2         Cswitch     241  Readch  1630.6K
                                                Syscall    2931  Writech   11660
CPU  User%  Kern%  Wait%  Idle%  Physc   Entc   Reads       247  Rawin         0
ALL    2.8    3.5    0.0   93.8   0.11    6.3   Writes       17  Ttyout      174
                                                Forks         5  Igets         0
Network  KBPS   I-Pack  O-Pack   KB-In  KB-Out  Execs         8  Namei       393
lo0       7.4      6.0     6.0     3.7     3.7  Runqueue    1.0  Dirblk        0
en0       0.5      2.5     1.5     0.1     0.3  Waitqueue   0.0
Disk    Busy%     KBPS     TPS KB-Read KB-Writ  PAGING           MEMORY
hdisk0    0.0    541.5    10.0   523.6    17.9  Faults     3118  Real,MB    5120
hdisk12   0.0    522.9     4.0   522.9     0.0  Steals        0  % Comp     24
hdisk8    0.0      0.7     1.0     0.7     0.0  PgspIn        0  % Noncomp   1
hdisk4    0.0      0.0     0.0     0.0     0.0  PgspOut       0  % Client    1
hdisk5    0.0      0.0     0.0     0.0     0.0  PageIn        0
hdisk9    0.0      0.0     0.0     0.0     0.0  PageOut       0  PAGING SPACE
hdisk1    0.0      0.0     0.0     0.0     0.0  Sios          0  Size,MB    6432
hdisk13   0.0      0.0     0.0     0.0     0.0                   % Used      0
                                                NFS (calls/sec)  % Free    100
Name            PID  CPU%  PgSp Owner           ServerV2       0
mysqld       258302   0.1 400.8 mysql           ClientV2       0   Press:
kpxagent     291036   0.1   7.7 root            ServerV3       0   "h" for help
topas        294954   0.0   1.3 root            ClientV3       0   "q" to quit
getty        278696   0.0   0.4 root
rmcd         344234   0.0   6.4 root
gil           86058   0.0   0.9 root
sshd         241810   0.0   0.9 root
rpc.lock     229508   0.0   1.2 root
sddsrv       282694   0.0   1.2 root
aixDataP     270550   0.0   9.0 root
aixmibd6     118930   0.0   1.1 root
syncd        127226   0.0   0.5 root

crosys

View Public Profile for crosys

Find all posts by crosys

10 More Discussions You Might Find Interesting

1. HP-UX

Copy on Write page faults

Hello Please can you tell me how to access COPY ON WRITE page faults in HP -UNIX. I found the structure in /usr/include/sys/vmmeter with the structure name vmmeter. Please tell me the function to fill the values to this structure and also the arguments for function.::

2. HP-UX

Intransient blocking page faults

Hi, Will anybody tell me what is this 'intransient blocking page faults' in HP-UX, it is in the structure _pst_vminfo in the header file /ust/include/sys/pstat/vm_pstat_body.h?

3. AIX

High Page Faults

Sorry my poor english In 570 pseries nmon shows excessive page faults, ascents of something more than 30000 Page faults. System: AIX 5.2 ML5 Processor Type: PowerPC_POWER5 Number Of Processors: 2 Processor Clock Speed: 1656 MHz CPU Type: 64-bit Kernel Type: 64-bit Memory Size: 2816 MB ...

4. AIX

How to Find if AIX machine is LPar or not

Hi, I want to know if there is any command which will tell me if an AIX machine is a LPar or not. I am using "lparstat -i" but it does not work on all AIX machines. Is there any generic command by whose output I will come to know if it is an LPar or not? Please help. Thanks, Vineet

5. AIX

Lots of page faults and free memory

Hello, I've been reading your forums for quite a while and the great amount of information I find here always come in hand.This time however, I need some specific help... I have a doubt with an AIX server which I'm failing to understand as I'm new to its concept of memory management... ...

6. AIX

AIX 7.1 high page faults

hi guys i hope you can help me with this situation. i have 2 lpar with aix 7.1 and oracle 11gr2 in grid mode. when i start nmon to check the current system health i notice that page fault are over 3000/s. than i have opened a case with ibm and they say that the problem is not paging nor...

7. AIX

Will it affect my AIX LPAR security, when i set up email alerts on AIX server.

Hello, I've set up email alerts on AIX Servers. so that i can get email notifications (via mail relay server) when ever there is abnormal behavior. for example 1) my script monitors CPU/disk/memory etc... when it reaches high water ark, it will send an email alert. 2) disk usage alerts 3)...

8. Solaris

Page faults on OS

Hi guys, I have a zone on a M5000 server running solaris 10. The zone has an SAP application running on it and facing some performance issues. As part of the troubleshooting, I've been recommended to look for any paging on the OS. Please advise how to look for the paging. I've been looking at...

9. AIX

High Paging when lots of free memory AIX 5.3

I am new to AIX, I have few AIX 5.3 servers and I could see there are significant difference in paging space utilization on servers even though they are running same applications below server is working fine which shows 2-5 % paging usage throuh out the day cpu_scale_memp = 8...

10. AIX

AIX LPAR FC connection to SAN

Hi all, In my system, I have HMC 7 with Power Machine 6 & 7. On the managed system, we have many lpars. In some lpars, I can see they are using virtual fiber channel to connect to DS8K storage. In search with google, I understand that it is configured with VIOS server to share the physical FC...

10 More Discussions You Might Find Interesting

1. HP-UX

Copy on Write page faults

Discussion started by: manjunath

2. HP-UX

Intransient blocking page faults

Discussion started by: sushaga

3. AIX

High Page Faults

Discussion started by: daviguez

4. AIX

How to Find if AIX machine is LPar or not

Discussion started by: vineetd

5. AIX

Lots of page faults and free memory

Discussion started by: flpgdt

6. AIX

AIX 7.1 high page faults

Discussion started by: gullio23

7. AIX

Will it affect my AIX LPAR security, when i set up email alerts on AIX server.

Discussion started by: System Admin 77

8. Solaris

Page faults on OS

Discussion started by: frum

9. AIX

High Paging when lots of free memory AIX 5.3

Discussion started by: bibish

10. AIX

AIX LPAR FC connection to SAN

Discussion started by: Phat