First off, i'd like to apologize for leaving some (unnoticed) typos in my first answer: vmo -o should have read vmo -a of course, analogous with schedo and ioo. I wanted to know the state of all your kernel tuning parameters and hence you to print out these values.

Still, you could have anticipated that by having a single look in the man page of one of these tools when the syntax error showed up. The fact that i am trying to help you does not exempt you from showing initiative of your own. When i give you a command with some explanation you might as well have a look in the documentation to understand what the command does so to be able to use it yourself the next time a similar problem shows up. You may even ask (here or in a separate thread) if something puzzles you and you are not able to figure it out yourself. But DO something, by all means! Not understanding something is OK, doing nothing is NOT.

OK, back to your problem. I suppose you have taken the data when the system was under load, not when the (main) application was switched or doing nothing. If it was not under load: all i wrote below is under the presumption of the system being under load and hence nonsense. Start over again and this time with real data, not watching the system doing nothing.

The output of iostat shows that there is next to no activity at all on the disks. The "grep" part of the command i gave you filters out all disks with 0.0 activity and in your case that left practically none. The minimal activity at the end is mainly the system disks, probably (re-)loading some program or library to memory. I'd say your disks are idle.

The vmstat output shows a similar picture: the first column ("r" - "run queue") shows a lot of programs running, but the system has all the means to run them: the second column ("b" - blocked processes) is constantly 0, so all the processes which could run are able to do so. The last 4 columns emphasize this. They are labeled "us", "sy", "id", "wa" and are percentages of the CPU usage. Together they add up to 100. "us" is for "user", the percentage of the time the CPU(s) spend(s) processing code of your programs. "sy" is for "system" and is the time the CPU spends executing system code, "id" is for "idle" - the CPU having nothing to do - and the last "wa" is for "wait": the time the CPU would be avle to run a process but has to wait because the process waits for I/O (usually swapping).

As you see the value for "wa" is 0 (this is how it should be), the value of "us" and "sy" are both very low and "id" is near 100. All in all your system looks like being overpowered a lot. Probably you could do with half or a quarter of the real CPUs assigned to this LPAR in the HMC profile.

The colums labeled "pi" and "po" count pages swapped in ("pi") and out ("po"). They are constantly 0, which is how it should be. At the same time this tells me that the system has now memory problem at all. This is further emphasized by the fact that "in" ("pages inspected - for possible swapping, that is) is low too - the system doesn't even bother to look for possibilities to make memory available, which means it has this resource in relative abundance.

The same story is told by the output of svmon which gives an aggregated, long-term account of memory consumption. Numbers there are memory pages (4k in AIX). Note the second number in the first line, "inuse", which is the memory available for programs: 7.8 million x 4k ~ 30G minus some system overhead (the difference between "inuse" and "size"). Now compare this with "virtual", the next to last value in the first line. This is the memory the system has really needed over time. 2.2 mio x 4k ~ 9G. This is what the system really needs to have, everything else is more "nice to have" than of real value. I would consider shrinking the system down to 12 GB (the necessary 9 plus some in contingency) if this is the typical status of the system.

Conclusion: if your system is slower than expected than this definitely has nothing to do with memory or CPU. It might be the network (we haven't investigated that yet), but frankly, i doubt that, because some residues of that would show up in the "vmstat -v" output.

Another possible reason to investigate would be sloppy application programming. As i don't know anything about the application you run this is pure speculation.

There could be some arbitrary shortage of assigned memory for the application: check with some DBA the size and the layout of the SGA and some related DB tuning options. DB_BLOCK_BUFFERS for instance, might put an arbitrary constraint in I/O operations. Similar, if some Java processes run on top of the database they might have not enough memory assigned. This is a common problem in Websphere application servers. The number of "pinned pages" in the output of "vmstat -v" suggests that the SGA is using only a small part of the available memory.

I hope this helps.

bakunin