Login or Register to Ask a Question and Join Our Community


AIX

Lots of page faults on AIX mySQL lpar

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems AIX Lots of page faults on AIX mySQL lpar
# 1  
Old 12-03-2010
Lots of page faults on AIX mySQL lpar
Hi,

OS = AIX 5.3

Large number of page faults recently start to occure on AIX 5.3 lpar with mysql database installed. I need help in setting AIX OS parameter to solve the paging problem and some guidance on interpreting my stats
t

Code:
Code:
 # vmstat -v──────────────────────────────────────────────────────────┘
              1310720 memory pages
              1247894 lruable pages
               636001 free pages
                    4 memory pools
               194085 pinned pages
                 80.0 maxpin percentage
                  5.0 minperm percentage
                 20.0 maxperm percentage
                  2.1 numperm percentage
                27349 file pages
                  0.0 compressed percentage
                    0 compressed pages
                  2.1 numclient percentage
                 20.0 maxclient percentage
                27349 client pages
                    0 remote pageouts scheduled
                  116 pending disk I/Os blocked with no pbuf
                    0 paging space I/Os blocked with no psbuf
                 2228 filesystem I/Os blocked with no fsbuf
                    0 client filesystem I/Os blocked with no fsbuf
                    0 external pager filesystem I/Os blocked with no fsbuf
                    0 Virtualized Partition Memory Page Faults
                 0.00 Time resolving virtualized partition memory page faults

Code:

Code:
mysql01 / # vmstat 5 5

System configuration: lcpu=8 mem=5120MB ent=1.80

kthr    memory              page              faults              cpu
----- ----------- ------------------------ ------------ -----------------------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
 1  0 641346 636011   0   0   0   0    0   0   8  852 176  1  0 99  0  0.03   1.4
 1  0 641346 636011   0   0   0   0    0   0   9   73 175  0  0 99  0  0.01   0.4
 1  0 641346 636011   0   0   0   0    0   0   0   32 154  0  0 99  0  0.00   0.3
 1  0 641344 636013   0   0   0   0    0   0   1  108 170  0  0 99  0  0.01   0.3
 1  0 641344 636013   0   0   0   0    0   0   0   28 152  0  0 99  0  0.01   0.3


memory and pagging
Code:

Code:
┌─topas_nmon──j=JFS──────────────Host=mysql01────────Refresh=2 secs───00:53.20─┐
│ Memory ──────────────────────────────────────────────────────────────────────│
│          Physical  PageSpace |        pages/sec  In     Out | FileSystemCache│
│% Used       51.5%      0.1%  | to Paging Space   0.0    0.0 | (numperm)  2.1%│
│% Free       48.5%     99.9%  | to File System    0.5    0.5 | Process   36.0%│
│MB Used    2637.5MB     7.7MB | Page Scans        0.0        | System    13.5%│
│MB Free    2482.5MB  6424.3MB | Page Cycles       0.0        | Free      48.5%│
│Total(MB)  5120.0MB  6432.0MB | Page Steals       0.0        |           -----│
│                              | Page Faults       0.5        | Total    100.0%│
│------------------------------------------------------------ | numclient  2.1%│
│Min/Maxperm     244MB(  5%)  975MB( 19%) <--% of RAM         | maxclient 19.0%│
│Min/Maxfree     960   1088       Total Virtual   11.3GB      | User      34.0%│
│Min/Maxpgahead    2      8    Accessed Virtual    2.4GB 21.7%| Pinned    14.8%│
│──────────────────────────────────────────────────────────────────────────────│

topas shost
Code:
Code:
Topas Monitor for host:    mysql01              EVENTS/QUEUES    FILE/TTY
Sat Dec  4 00:59:25 2010   Interval:  2         Cswitch     237  Readch      245
                                                Syscall     176  Writech  848.8K
CPU  User%  Kern%  Wait%  Idle%  Physc   Entc   Reads         1  Rawin         0
ALL    0.4    0.2    0.0   99.4   0.01    0.7   Writes       29  Ttyout      239
                                                Forks         0  Igets         0
Network  KBPS   I-Pack  O-Pack   KB-In  KB-Out  Execs         0  Namei         0
en0       0.4      0.5     1.0     0.0     0.3  Runqueue    1.0  Dirblk        0
lo0       0.0      0.0     0.0     0.0     0.0  Waitqueue   0.0

Disk    Busy%     KBPS     TPS KB-Read KB-Writ  PAGING           MEMORY
hdisk12   0.0    616.0    14.0     0.0   616.0  Faults        0  Real,MB    5120
hdisk8    0.0    236.0    15.0     2.0   234.0  Steals        0  % Comp     49
hdisk5    0.0      0.0     0.0     0.0     0.0  PgspIn        0  % Noncomp   2
hdisk4    0.0      0.0     0.0     0.0     0.0  PgspOut       0  % Client    2
hdisk0    0.0      0.0     0.0     0.0     0.0  PageIn        0
hdisk1    0.0      0.0     0.0     0.0     0.0  PageOut       0  PAGING SPACE
hdisk9    0.0      0.0     0.0     0.0     0.0  Sios          1  Size,MB    6432
hdisk13   0.0      0.0     0.0     0.0     0.0                   % Used      0
                                                NFS (calls/sec)  % Free    100
Name            PID  CPU%  PgSp Owner           ServerV2       0
mysqld       454808   0.4 1475.7 mysql          ClientV2       0   Press:
topas        442526   0.0   1.3 root            ServerV3       0   "h" for help
getty        159860   0.0   0.4 root            ClientV3       0   "q" to quit
gil           86058   0.0   0.9 root
xmgc          49176   0.0   0.4 root
sshd         323710   0.0   0.9 root
sddsrv       315578   0.0   1.5 root
sendmail     180318   0.0   1.0 root
rpc.lock     168130   0.0   1.2 root
kpxagent     299174   0.0   8.5 root
syncd        127102   0.0   0.5 root
netm          81960   0.0   0.4 root
j2pg         106562   0.0   3.5 root
aixDataP     237806   0.0   9.2 root
rmcd         340146   0.0   6.3 root
topasrec     262374   0.0   1.0 root
IBM.CSMA     241670   0.0   2.2 root
lvmbb        122964   0.0   0.4 root
psmd          24588   0.0   0.8 root
hostmibd     131224   0.0   0.9 root

Thanks in advance
# 2  
Old 12-03-2010
took a quick look at the data you provided. there are no issues with memory. are you actually experiencing performance issues?
# 3  
Old 12-04-2010
page faults are normal AIX behaviour ... content is written into memory - that is NORMAL behaviour and NOT a problem. It is just how VMM works:
AIX maps pages into real memory based on demand. When an application references a page that is not mapped into real memory, the system generates a page fault. To resolve the page fault, the AIX kernel loads the referenced page to a location in real memory. If the referenced page is a new page (that is, a page in a data heap of the process that has never been previously referenced), "loading" the referenced page simply means filling a real memory location with zeros (that is, providing a zero-filled page). If the referenced page is a pre-existing page (that is, a page in a file or a previously paged out page), loading the referenced page involves reading the page from the disk (paging space or disk file system) into a location in real memory.

Once a page is loaded into real memory, it is marked as unmodified. If a process or the kernel modifies the page, the state of the page changes to modified. This allows AIX to keep track of whether a page has been modified after it was loaded into memory.

Please read more here

Apart from that your system is completely untuned. You should at least tune your lru_file_repage to 0 or your system will start swapping sooner or later what will cause REAL problems and your minperm/maxperm values are as well far away from any recommended values ...


Regards
zxmaus
# 4  
Old 12-04-2010
Hi,

Thanks on fast replay. My colleague started to complain that system is suddenly become very slow. I did some nmon stats and find out that system is generating more than 13000 I/O in sec and lots of page faults. I did some adjustments on AIX OS;

Code:
vmo -p -o minperm%=3
vmo -p -o maxperm%=90
vmo -p -o maxclient%=90
vmo -p -o minfree=960
vmo -p -o maxfree=1088
vmo -p -o lru_file_repage=0
ioo -p -o pv_min_pbuf=1024

I also edit /etc/filesystems and use 'dio' because AIX lpar is attached to SAN so I want to take advantage of caching, throughput optimization, and performance by enabling file system direct I/O on AIX. This, according to my knowledge, will allow the file system buffer cache to be bypassed which eliminates the double buffering of data with mySQL which can adversely affect file system I/O performance. I tried to upload excel zip file of nmon statistic but error always appeard (file is less then 3 MB). Problem still remain, slow performance.

Some new stats are:

Code:
 # vmstat -s──────────────────────────────────────────────────────────┘
               788510 total address trans. faults
                49725 page ins
                11898 page outs
                    0 paging space page ins
                    0 paging space page outs
                    0 total reclaims
               440936 zero filled pages faults
                 4617 executable filled pages faults
                    0 pages examined by clock
                    0 revolutions of the clock hand
                    0 pages freed by the clock
                60589 backtracks
                    0 free frame waits
                    0 extend XPT waits
                19289 pending I/O waits
                61623 start I/Os
                32640 iodones
               133924 cpu context switches
                35253 device interrupts
                16549 software interrupts
                78787 decrementer interrupts
                 1491 mpc-sent interrupts
                 1491 mpc-received interrupts
                  300 phantom interrupts
                    0 traps
               958258 syscalls

Code:
 # vmstat 5 5─────────────────────────────────────────────────────────┘
System configuration: lcpu=8 mem=5120MB ent=1.80
kthr    memory              page              faults              cpu
----- ----------- ------------------------ ------------ -----------------------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
 1  0 314163 966357   0   0   0   0    0   0   2  210 170  0  0 99  0  0.01   0.3
 1  0 314163 966357   0   0   0   0    0   0   0   30 159  0  0 99  0  0.01   0.3
 1  0 314172 966348   0   0   0   0    0   0   1  364 165  1  1 98  0  0.04   2.3
 1  0 314172 966348   0   0   0   0    0   0  47  138 262  0  0 99  0  0.01   0.5
 1  1 314228 966291   0   0   0   0    0   0 311 1517 846  1  1 97  1  0.05   2.5

Code:
Topas Monitor for host:    mysql01              EVENTS/QUEUES    FILE/TTY
Sat Dec  4 17:15:49 2010   Interval:  2         Cswitch     241  Readch  1630.6K
                                                Syscall    2931  Writech   11660
CPU  User%  Kern%  Wait%  Idle%  Physc   Entc   Reads       247  Rawin         0
ALL    2.8    3.5    0.0   93.8   0.11    6.3   Writes       17  Ttyout      174
                                                Forks         5  Igets         0
Network  KBPS   I-Pack  O-Pack   KB-In  KB-Out  Execs         8  Namei       393
lo0       7.4      6.0     6.0     3.7     3.7  Runqueue    1.0  Dirblk        0
en0       0.5      2.5     1.5     0.1     0.3  Waitqueue   0.0
Disk    Busy%     KBPS     TPS KB-Read KB-Writ  PAGING           MEMORY
hdisk0    0.0    541.5    10.0   523.6    17.9  Faults     3118  Real,MB    5120
hdisk12   0.0    522.9     4.0   522.9     0.0  Steals        0  % Comp     24
hdisk8    0.0      0.7     1.0     0.7     0.0  PgspIn        0  % Noncomp   1
hdisk4    0.0      0.0     0.0     0.0     0.0  PgspOut       0  % Client    1
hdisk5    0.0      0.0     0.0     0.0     0.0  PageIn        0
hdisk9    0.0      0.0     0.0     0.0     0.0  PageOut       0  PAGING SPACE
hdisk1    0.0      0.0     0.0     0.0     0.0  Sios          0  Size,MB    6432
hdisk13   0.0      0.0     0.0     0.0     0.0                   % Used      0
                                                NFS (calls/sec)  % Free    100
Name            PID  CPU%  PgSp Owner           ServerV2       0
mysqld       258302   0.1 400.8 mysql           ClientV2       0   Press:
kpxagent     291036   0.1   7.7 root            ServerV3       0   "h" for help
topas        294954   0.0   1.3 root            ClientV3       0   "q" to quit
getty        278696   0.0   0.4 root
rmcd         344234   0.0   6.4 root
gil           86058   0.0   0.9 root
sshd         241810   0.0   0.9 root
rpc.lock     229508   0.0   1.2 root
sddsrv       282694   0.0   1.2 root
aixDataP     270550   0.0   9.0 root
aixmibd6     118930   0.0   1.1 root
syncd        127226   0.0   0.5 root

# 5  
Old 12-04-2010
assuming that you run jfs2 and not jfs filesystems, you should rather choose cio than dio to get the full advantage of the settings. You might want to set AIXTHREADSCOPE=S in your /etc/environment and you want for sure mount your filesystems with noatime option. I do not know anything about mysql on AIX but if the DB is capable to take advantage from async IO like oracle does, than you may want to adjust the aio settings to a higher maxreq setting and to give you more hints on optimization, it would be interesting to know what kind of workload your DB has ... random like for trading / oltp databases or more serial ... like typical for batch- and reporting workloads.

BTW - if you want to show us sample data - please take that data from a time where your system is particularly busy - the data you have posted so far shows a completely idle box - and a system that doesnt do anything has usually no performance issues.

You have a realtively high amount of cpu for a pretty low amount of memory - if your DB has caches like oracle or sybase, it might be worth extending these for more performance ...

Regards
zxmaus
Last edited by zxmaus; 12-04-2010 at 10:08 PM..
# 6  
Old 12-05-2010
Hi,

Thanks Zxmaus,

Your recomedation to use cio insted dio helps a lot. System looks mutch better know.

Best regards
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. AIX

AIX LPAR FC connection to SAN

Hi all, In my system, I have HMC 7 with Power Machine 6 & 7. On the managed system, we have many lpars. In some lpars, I can see they are using virtual fiber channel to connect to DS8K storage. In search with google, I understand that it is configured with VIOS server to share the physical FC... (7 Replies)
Discussion started by: Phat
7 Replies

2. AIX

High Paging when lots of free memory AIX 5.3

I am new to AIX, I have few AIX 5.3 servers and I could see there are significant difference in paging space utilization on servers even though they are running same applications below server is working fine which shows 2-5 % paging usage throuh out the day cpu_scale_memp = 8... (12 Replies)
Discussion started by: bibish
12 Replies

3. Solaris

Page faults on OS

Hi guys, I have a zone on a M5000 server running solaris 10. The zone has an SAP application running on it and facing some performance issues. As part of the troubleshooting, I've been recommended to look for any paging on the OS. Please advise how to look for the paging. I've been looking at... (4 Replies)
Discussion started by: frum
4 Replies

4. AIX

Will it affect my AIX LPAR security, when i set up email alerts on AIX server.

Hello, I've set up email alerts on AIX Servers. so that i can get email notifications (via mail relay server) when ever there is abnormal behavior. for example 1) my script monitors CPU/disk/memory etc... when it reaches high water ark, it will send an email alert. 2) disk usage alerts 3)... (5 Replies)
Discussion started by: System Admin 77
5 Replies

5. AIX

AIX 7.1 high page faults

hi guys i hope you can help me with this situation. i have 2 lpar with aix 7.1 and oracle 11gr2 in grid mode. when i start nmon to check the current system health i notice that page fault are over 3000/s. than i have opened a case with ibm and they say that the problem is not paging nor... (10 Replies)
Discussion started by: gullio23
10 Replies

6. AIX

Lots of page faults and free memory

Hello, I've been reading your forums for quite a while and the great amount of information I find here always come in hand.This time however, I need some specific help... I have a doubt with an AIX server which I'm failing to understand as I'm new to its concept of memory management... ... (8 Replies)
Discussion started by: flpgdt
8 Replies

7. AIX

How to Find if AIX machine is LPar or not

Hi, I want to know if there is any command which will tell me if an AIX machine is a LPar or not. I am using "lparstat -i" but it does not work on all AIX machines. Is there any generic command by whose output I will come to know if it is an LPar or not? Please help. Thanks, Vineet (3 Replies)
Discussion started by: vineetd
3 Replies

8. AIX

High Page Faults

Sorry my poor english In 570 pseries nmon shows excessive page faults, ascents of something more than 30000 Page faults. System: AIX 5.2 ML5 Processor Type: PowerPC_POWER5 Number Of Processors: 2 Processor Clock Speed: 1656 MHz CPU Type: 64-bit Kernel Type: 64-bit Memory Size: 2816 MB ... (1 Reply)
Discussion started by: daviguez
1 Replies

9. HP-UX

Intransient blocking page faults

Hi, Will anybody tell me what is this 'intransient blocking page faults' in HP-UX, it is in the structure _pst_vminfo in the header file /ust/include/sys/pstat/vm_pstat_body.h? (4 Replies)
Discussion started by: sushaga
4 Replies

10. HP-UX

Copy on Write page faults

Hello Please can you tell me how to access COPY ON WRITE page faults in HP -UNIX. I found the structure in /usr/include/sys/vmmeter with the structure name vmmeter. Please tell me the function to fill the values to this structure and also the arguments for function.:: (5 Replies)
Discussion started by: manjunath
5 Replies
Login or Register to Ask a Question