I would like to hear your thoughts about this. We are running our Data warehouse on DB2 DPF (partition environment) and I have notice that sometimes we hit the Asynchronous-I/O-Processes peak. DB2 relies heavily on Asynchronous I/O so I would believe this has an negative impact.We are currently running AIX 5.3. I wonder if this constraint would impact the performance of the ETL flow and any other activities on this server. I need to convince our Unix Administrators to increase this parameter but I need a more strong and valid argument. For those more skillfull with AIX and performance please if you can provide me your comments would be highly appreciated. thanks...
Well, too many processes just waste time in swap and longer run times. Once you use up the limiting resource, more processes may lower throughput and increase latency. It is like if two people show up to dry 2 loads of clothes, and only 2 dryers are open. If one guy says, go ahead, use both, he is not delayed and the other guy is there half the time. Overloads belong in queues. Red lights on downramps keep speeds up. Maybe it will run faster if you lower it. You need precise measurements and then you can move it a little at a time, to see. http://www.aixexpert.com/wiki/index...._for_AIX_4_3_3
Thanks for your answer. I like your analogy but I thought that in my case dealing with a big Database server (5 TB data warehouse database ) having extra Async I/O processes would increase/help performance. What worries me is I'm executing the isotat -AQ command and I'm getting :
Before increasing the maxservers, you could try if the behaviour is getting better when increasing the maxreqs 1st. On AIX 5.3 it is per default 4096. In AIX 6.1 they start with 16384 for example. If you had hit the max number of async I/O requests, you will most probably have gotten a Oracle DB error with a crash (see this: What is maxreqs parameter in AIX?)
What you see with iostat -AQ is just the average since boot. If you want an actual size of the ascync I/O queue, you might want to add some interval parameters like iostat -AQ 2 which will report every 2 seconds.
The number of active AIO servers processes you see with a ps output for example, is just the number of servers, that had been started at some time of a peak. It is not necessary the number of servers that are actually doing some work. To see these, you could use topas -i2. Here an example from a busy Oracle box:
Code:
┌─topas_nmon──S=WLMsubclasses────Host=somehost──────Refresh=2 secs───11:08.03─────────────────────────────────────────────────────────┐
│ Asynchronous-I/O-Processes ──────────────────────────────────────────────────────────────────────────────────────────────────────────│
│ Total AIO processes=1100 Actually in use= 59 CPU used= 20.9% [Use zero │
│ All time peak=1100 Recent peak= 59 Peak= 20.9% to reset] │
There are currently 1100 servers started, but now at production times, there are in this 2 seconds just 59 of them really active.
When aioserver processes are not doing any work anymore, they stay started and so their number does not decrease. Their number stays at their peak value since boot, which is here 1100. We have maxservers=50, so 50 for every logical cpu (lcpu) in the box, maxreqs=32768 (just in case) and minservers=5 to have a buffer for initial workload. If more are needed, the box just starts more of them.
It could be that in some special job or night job etc. there is much more demand of async IO, so that could be the reason, the box started all of the 1100.
Depending on what your current values for AIO are (post a lsattr -El aio0, it could no bad idea to increase them and monitor the box. For long term monitoring, write a script or maybe use something like nmon.
Thanks for you valuable comments. Here is more details about my config.
The below are our AIO settings:
Quote:
$ lsattr -El aio0
autoconfig available STATE to be configured at system restart True
fastpath enable State of fast path True
kprocprio 39 Server PRIORITY True
maxreqs 4096 Maximum number of REQUESTS True
maxservers 10 MAXIMUM number of servers per cpu True
minservers 1 MINIMUM number of servers True
And below is a snapshot from nmon/topas
Code:
┌─topas_nmon──S=WLMsubclasses────Host=uapmwh02───────Refresh=2 secs───16:57.47──────────────────────────────────────────────┐
│ Asynchronous-I/O-Processes ───────────────────────────────────────────────────────────────────────────────────────────────│
│ Total AIO processes= 80 Actually in use= 0 CPU used= 1.5% [Use zero │
│ All time peak= 80 Recent peak= 6 Peak= 1.9% to reset] │
│ Top-Processes-(240) ─────Mode=3 [1=Basic 2=CPU 3=Perf 4=Size 5=I/O 6=Cmds]───────────────────────────────────────────────│
│ PID %CPU Size Res Res Res Char RAM Paging
Also we are on smt so the config is:
System configuration: lcpu=8 ent=1.50
So in our case we get 80 AIO servers processes. This number seems to be a bit low for a DB server performing a large number of DB prefetches.
I can see in topas reaching that peak very often but is not enough to justify a chance according to our Unix Administrators.
I think it is a very good reason, as maxservers=10 is just a default value which is usually not sufficient for a box running a DB that is taking advantage of async I/O, as well as all other settings for your aio. Your admins should read any of the freely available tuning guides for Oracle on AIX and other useful tuning guides for AIX on the web.
I strongly recommend following values as a start:
Code:
maxreqs=16384
maxservers=50
minservers=5
The values of the queue and the number of maximum started AIO servers should be monitored and tuned accordingly. But there is no doubt, that your DB will have an easier live when it is given what it needs and supports.
On the other hand, ask your admins why it should not be done? It does not damage anything nor does it waste precious amounts of RAM. In fact it is a simple tuning task which those that are responsible for performance monitoring and tuning could/should have noticed or at least be thankful that you noticed them.
Quote:
I can see in topas reaching that peak very often but is not enough to justify a chance according to our Unix Administrators.
Running your critical business applications at best possible performance with the given hardware and software should be their interesst too, especially when it involves so little effort to accomplish.
Last edited by zaxxon; 04-19-2011 at 05:30 AM..
Reason: typos and phrasing
Hi,
I'm supposed to capture many performance stats on AIX 6 and stuck up with below:
Priority queue
Disk cache hit%
Page out rate
Swap out rate
Memory queue
I see vmstatis helpful for "page out" but not sure how to get the "rate".
Could anyone please let me know how to get these... (4 Replies)
Hello,
I encounter some performance issues on my AIX 5.3 server running in a LPAR on a P520. How do I investigate performance issues in AIX. Is there any kind of procedure that takes me to the steps to investigate my server and find the sub systems that is causing the issues?
The performance... (1 Reply)
Hello
I am new user of AIX; I have only basic knowledge of the UNIX commands, and I want to create script that will monitor the performance and resources usage on AIX 6.1 machine.
Basically I wan to start a loop that will grab, every 10 seconds, the CPU usage, the memory usage, the disk usage,... (1 Reply)
Hi Guys,
This is the situation I am in. Provide your views and input where should I start?
I have one P7 test server and a p520 production server. the job is taking pretty long on the P7 test server when compared to the P5 production server. below is the full detail.
Informix... (5 Replies)
Gurus, i have process that runs 5 times a day.
it runs normally (takes about 1 hour) to complete in 3 runs
but it is takes about ( 3 hrs to complete) two times
So i need to figure out why it takes significanlty high time during
those 2 runs.
The process is a shell script that connect to... (2 Replies)
I am trying to analyze the performance of an AIX system. I think I may have a disk I/O issue, but I am asking for help to validate or invalidate this assumption. I ran the commands below during a period of peak load.
Please help me to find any performance bottlenecks. Thanks in advance for your... (15 Replies)
I'm doing performance testing for one application which works on AIX.
But I don't know which performance parameters of memory need to be collected. Now, I just know very few:
1. page in
2. page out
3. fre
They are all collected by "vmstat" command.
I want to know, except for above... (2 Replies)
Hiya all,
I am a newbie sysadmin to AIX, i have worked on HPUX for 3 years.
I have started a new role with in an IBM house and because there is me and one other there are a couple of issues I cannot work out:
We havehad a production server slowing down processing batch jbs over the past... (6 Replies)
Hello,
I have a performance problem on an AIX box. I'm not sure what is causing this and hoping someone may have suggestions. Currently I'm noticing that cpu's are waiting while the box is in a idle state. I checked the disks and none of them are at 100%. If they were then I would understand... (3 Replies)
04-12-2011
