First off I'd like to state that your data is not nearly complete. As this is a welcome occasion to go through a performance tuning procedure lets cover this with a little depth:

1. Before you begin: the SLA

The first and most vital thing about performance tuning is to get to an SLA, a service level agreement, with the customer. It doesn't matter if "customer" is a real customer or just another department down the hall - there has to be some agreement between systems administration and user about how fast is fast enough. Otherwise it will be one of these endless (and pointless) tuning orgies which leaves everybody involved only unsatisfied - and yet exhausted - after a lot of work.

Agree with your customer about some measureable fact and declare the tuning to be successfully once this limit is reached. Something like "the system has to do X transactions per second" or "the response time of this program should not exceed 2 seconds" or "the depth of this queue should not exceeed X entries" or "the system should process X GB of data per hour", etc., etc..

Don't forget: make it written! Users tend to "forget" what they have agreed to, so get a written statement agreed upon by all parties involved. In case you wondered: "measureable" means "countable". It could be a wee bit faster sometimes is NOT "measureable", its daydreaming - your job is not to make dreams come true, yes?

2. Get your data

The next step is to get the data - ALL data. First you need the unchanging things: machine specifications, software releases, configurations, customization, etc., etc., etc.

The following is AIX-specific, but you should easily be able to "port" this to other OSes. Also notice that the lists are incomplete in nature: add to them whatever seems to add to the picture if necessity arises:

a) Hardware:
prtconf output

b) Software releases
instfix -i | grep AIX_ML (or TL)
version information of the application program(s) in question

c) customization
ioo -a output
vmo -a output
schedo -a output
no -a output
lsps -a output
crontabs


3. Get your data - again

After this, analyze the machine in light of what the customers tell you. In which regard is it "slow" - bad I/O? slow disks? unresponsive network connections? Write that down and save it for future reference.

Only now get the real performance data. A good start is (again, this is AIX-minded, but could easily be translated to other OS flavours):

vmstat
iostat
netstat/entstat
svmon
lsps
ps


4. The tuning process

Only now the real tuning starts. Note that this is a repetitive process and be prepared to go over step 3 & 4 again and again. Take the data gathered in step 3 and analyze them. Create a theory what is causing which symptom. (Btw.: everything can be a symptom. If the machine is responding notably faster for 10 minutes and then slows down again you want to know why this happens.) Look out for any repeting pattern in the data. If you find something try to find an explanation for it. That doesn't necessarily have to mean you could change it, but it will further you understanding of the systems workings.

Once you have a theory (explanation) of what happens why put this theory to test: apply - CAREFULLY! - selected changes to the system and watch what happens (basically go back to step 3, then compare).

Be sure to make only one change at a time. Otherwise you won't know which change has caused which difference in the data. You can tune only the same way you walk: one step after the other. If you try to make more the one step at the same time chances are you just jump on one foot up and down, effectively getting nowhere.

I hope this helps.

bakunin

PS: It has taken me some time to write this and in the meantime you have already gotten very good advice, so i have deleted what i have written about your actual problem. Still i think that talking about the tuning process in general is a good idea which is why i wrote this article. I really do hope it helps.