AIX 5.2 performance question

jhall · #1 (**permalink**) 11-04-2008

I am trying to analyze the performance of an AIX system. I think I may have a disk I/O issue, but I am asking for help to validate or invalidate this assumption. I ran the commands below during a period of peak load.

Please help me to find any performance bottlenecks. Thanks in advance for your help! Let me know if any more information is required since I am new to performance tuning on AIX.

*************************
vmstat 1 15
*************************

Code:

kthr     memory             page              faults        cpu
----- ----------- ------------------------ ------------ -----------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa
 1  1 203292   127   0   0   0 134  248   0 395 97470 530  8  8 79  5
 2  2 203007  1848   0   4   0 311  490   0 2469 16936 942  0  8  3 88
 0  3 203007  1179   0   0   0   0    0   0 2488 16406 769  1  3  2 94
 0  3 203007   492   0   0   0   0    0   0 2400 8130 817  2  2  1 94
 0  3 203007   131   0   0   0 831 1348   0 2390 8121 838  0  4  8 87
 0  3 203007   120   0   0   0 1075 1747   0 2313 16763 865  2  7  0 91
 0  2 203007   125   0   0   0 154  232   0 2393 6932 1163  0  8  6 86
 1  3 203007   130   0   0   0 435  696   0 2434 5414 881  0  6  0 93
 0  3 203007   124   0   0  14 539  858   0 2367 7972 787  0  7  6 86
 0  2 203007   130   0   0   0 1238 1946   0 2507 5370 866  0  5 12 83
 0  2 203007   133   0   0   0 1230 2034   0 2492 16579 817  3  4 16 77
 0  2 203007   128   0   0   0 1114 2043   0 2358 5366 849  0  9 11 80
 0  2 203007   129   0   0   0 1972 3248   0 2653 6180 976  2  4 14 82
 0  2 203007   133   0   0   0 1628 2781   0 2337 8640 875  0  5  5 90
 1  2 203007   132   0   0   0 1569 2571   0 2437 6540 1041  0  6  2 91

*************************
iostat 1 10
*************************

Code:

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          1.4        110.6               8.4      8.0       78.7       4.9

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0           0.3      15.4       0.7   49940629  30144516
hdisk3           0.0       2.3       0.1    8500120   3403880
hdisk1           0.4      23.6       0.9   50158859  72036249
hdisk2           0.0       0.1       0.0     286642     52624

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          1.0        1944.0               1.0      5.0       18.5      75.5

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0         100.0     8736.0     273.0       4352      4384
hdisk3           0.0       0.0       0.0          0         0
hdisk1         100.0     672.0     168.0         20       652
hdisk2           0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0          0.0               1.5      5.5        7.5      85.5

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0         100.0     8420.0     264.0       4224      4196
hdisk3           0.0       0.0       0.0          0         0
hdisk1          99.0     680.0     167.0          0       680
hdisk2           0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          1.0          4.0               1.5      6.0        1.5      91.0

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0         100.0     8704.0     272.0       4352      4352
hdisk3           0.0       0.0       0.0          0         0
hdisk1          98.0     700.0     163.0          0       700
hdisk2           0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0          0.0               0.0      6.5        5.0      88.5

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0         100.0     8736.0     273.0       4352      4384
hdisk3           0.0       0.0       0.0          0         0
hdisk1         100.0     664.0     166.0          0       664
hdisk2           0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0          0.0               0.5      3.5        0.0      96.0

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0         100.0     8832.0     276.0       4416      4416
hdisk3           0.0       0.0       0.0          0         0
hdisk1         100.0     676.0     166.0          0       676
hdisk2           0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0          0.0               2.0      8.5       17.0      72.5

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0         100.0     8576.0     268.0       4288      4288
hdisk3           0.0       0.0       0.0          0         0
hdisk1          93.0     632.0     155.0          0       632
hdisk2           0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0          0.0               0.0      3.5       10.0      86.5

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0         100.0     8164.0     265.0       4100      4064
hdisk3           0.0       0.0       0.0          0         0
hdisk1          99.0     664.0     166.0          0       664
hdisk2           0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0          0.0               1.5      7.5        6.5      84.5

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          99.0     7488.0     242.0       3744      3744
hdisk3           0.0       0.0       0.0          0         0
hdisk1          96.0     688.0     160.0          0       688
hdisk2           0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0          0.0               0.5      6.0        4.5      89.1

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          99.5     8278.6     261.7       4160      4160
hdisk3           8.0     131.3      28.9          0       132
hdisk1          99.5     632.8     157.2         12       624
hdisk2           0.0       0.0       0.0          0         0

*************************
lsps -s
*************************
Total Paging Space Percent Used
512MB 13%

nivaspIND · #2 (**permalink**) 11-04-2008

Even I am new but trying to help you

Have you checked the stats of lvmstat command .. we can find out which volumes or lv's are taking up more resources or as you said I/O requests in the peak time.

Nivas p5

shockneck · #3 (**permalink**) 11-04-2008

Quote:

Originally Posted by jhall

I am trying to analyze the performance of an AIX system. I think I may have a disk I/O issue[...]

So you want to hear that you have an I/O problem?

How To Ask Questions The Smart Way

Quote:

Originally Posted by jhall

*************************
vmstat 1 15
*************************
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 1 203292 127 0 0 0 134 248 0 395 97470 530 8 8 79 5
2 2 203007 1848 0 4 0 311 490 0 2469 16936 942 0 8 3 88
0 3 203007 1179 0 0 0 0 0 0 2488 16406 769 1 3 2 94
0 3 203007 492 0 0 0 0 0 0 2400 8130 817 2 2 1 94
0 3 203007 131 0 0 0 831 1348 0 2390 8121 838 0 4 8 87
0 3 203007 120 0 0 0 1075 1747 0 2313 16763 865 2 7 0 91
0 2 203007 125 0 0 0 154 232 0 2393 6932 1163 0 8 6 86
1 3 203007 130 0 0 0 435 696 0 2434 5414 881 0 6 0 93
0 3 203007 124 0 0 14 539 858 0 2367 7972 787 0 7 6 86
0 2 203007 130 0 0 0 1238 1946 0 2507 5370 866 0 5 12 83
0 2 203007 133 0 0 0 1230 2034 0 2492 16579 817 3 4 16 77
[code][...]

There is paging into the rootvg's filesystem while both run and blocked queue are small and a high wait at the same time. No paging space involved. This indicates a problem with the amount of computational memory compared to filecache memory rather than with I/O. Could you post the output of

Code:

# svmon -G
# vmo -a | egrep -i 'perm|cli|free|lru'

zaxxon · #4 (**permalink**) 11-04-2008

Additionally in your iostat you can see that hdisk0 and hdisk1 are heavily busy. They are about ~90-100% busy all the time while you took the measurement.

Quote:

tty: tin tout avg-cpu: % user % sys % idle % iowait
1.0 4.0 1.5 6.0 1.5 91.0

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 100.0 8704.0 272.0 4352 4352
hdisk3 0.0 0.0 0.0 0 0
hdisk1 98.0 700.0 163.0 0 700
hdisk2 0.0 0.0 0.0 0 0

It seems you have something on these disks which should be on separate disks. As Shockneck assumes too, that hdisk0 and hdisk1 belong to your rootvg, I'd say you check what kind of application data is placed in some LV there and move it to other disks to not hinder your rootvg.

Have a lspv to check what VGs are on hdisk0 and hdisk1, then have a lsvg -l on that VG to check what FS'es are there and try to move the stuff to disks that are not that busy or to new disks.

Then capture some seconds while this problem is occuring with filemon:

Code:

filemon -v -o fmon.out -O all

Stop this with "trcstop" after some 20-30 seconds.

You can check the file with

Code:

filemon -i fmon.out | more

You should see which LV etc. is the most busy to be sure what is making trouble there.

jhall · #5 (**permalink**) 11-05-2008

Thank you for your replies. The system is only this busy during the nightly processing. Otherwise, it is mostly idle.

I do know that the volume group, logical volume, and filesystem configuration is far from ideal. The entire system is running from local disks. Some of the application filesystems are located in the rootvg. I am just trying to figure out the best course of action to fix these issues.

I will capture the output of the commands above during tomorrow's processing and post the results.

Thanks again!

funksen · #6 (**permalink**) 11-05-2008

if it's really paging, try to create a striped or mirrored, depends on the vg design on hdisk2 hdisk3, paging space on hdisk 2 hdisk3, to balance the ps load on 4 disks instead of 1 or 2

mkps -a -n -s PPsyoulike vgonhdisk2/3 hdisk2

the size depends on the overall memory and the application you use

an pls run

Code:

lsvg
lsvg -l vgname

for every vg

Code:

lvmstat -v rootvg -e 

in the evening

and 
lvmstat -v rootvg -F
lvmstat -v rootvg -d

post the output here

bakunin · #7 (**permalink**) 11-05-2008

First off I'd like to state that your data is not nearly complete. As this is a welcome occasion to go through a performance tuning procedure lets cover this with a little depth:

1. Before you begin: the SLA

The first and most vital thing about performance tuning is to get to an SLA, a service level agreement, with the customer. It doesn't matter if "customer" is a real customer or just another department down the hall - there has to be some agreement between systems administration and user about how fast is fast enough. Otherwise it will be one of these endless (and pointless) tuning orgies which leaves everybody involved only unsatisfied - and yet exhausted - after a lot of work.

Agree with your customer about some measureable fact and declare the tuning to be successfully once this limit is reached. Something like "the system has to do X transactions per second" or "the response time of this program should not exceed 2 seconds" or "the depth of this queue should not exceeed X entries" or "the system should process X GB of data per hour", etc., etc..

Don't forget: make it written! Users tend to "forget" what they have agreed to, so get a written statement agreed upon by all parties involved. In case you wondered: "measureable" means "countable". It could be a wee bit faster sometimes is NOT "measureable", its daydreaming - your job is not to make dreams come true, yes?

2. Get your data

The next step is to get the data - ALL data. First you need the unchanging things: machine specifications, software releases, configurations, customization, etc., etc., etc.

The following is AIX-specific, but you should easily be able to "port" this to other OSes. Also notice that the lists are incomplete in nature: add to them whatever seems to add to the picture if necessity arises:

a) Hardware:
prtconf output

b) Software releases
instfix -i | grep AIX_ML (or TL)
version information of the application program(s) in question

c) customization
ioo -a output
vmo -a output
schedo -a output
no -a output
lsps -a output
crontabs

3. Get your data - again

After this, analyze the machine in light of what the customers tell you. In which regard is it "slow" - bad I/O? slow disks? unresponsive network connections? Write that down and save it for future reference.

Only now get the real performance data. A good start is (again, this is AIX-minded, but could easily be translated to other OS flavours):

vmstat
iostat
netstat/entstat
svmon
lsps
ps

4. The tuning process

Only now the real tuning starts. Note that this is a repetitive process and be prepared to go over step 3 & 4 again and again. Take the data gathered in step 3 and analyze them. Create a theory what is causing which symptom. (Btw.: everything can be a symptom. If the machine is responding notably faster for 10 minutes and then slows down again you want to know why this happens.) Look out for any repeting pattern in the data. If you find something try to find an explanation for it. That doesn't necessarily have to mean you could change it, but it will further you understanding of the systems workings.

Once you have a theory (explanation) of what happens why put this theory to test: apply - CAREFULLY! - selected changes to the system and watch what happens (basically go back to step 3, then compare).

Be sure to make only one change at a time. Otherwise you won't know which change has caused which difference in the data. You can tune only the same way you walk: one step after the other. If you try to make more the one step at the same time chances are you just jump on one foot up and down, effectively getting nowhere.

I hope this helps.

bakunin

PS: It has taken me some time to write this and in the meantime you have already gotten very good advice, so i have deleted what i have written about your actual problem. Still i think that talking about the tuning process in general is a good idea which is why i wrote this article. I really do hope it helps.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Announcing collectl - new performance linux performance monitor	MarkSeger	News, Links, Events and Announcements	0	10-26-2007 07:14 PM
pthread performance question	ldarden	AIX	1	03-18-2007 06:54 PM
I/O performance	gfhgfnhhn	UNIX for Advanced & Expert Users	2	09-10-2006 02:10 PM
Korn Shell Coprocess Performance Question	Mark Puddephat	Shell Programming and Scripting	8	12-14-2005 05:33 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread: