AIX

Hello,

I have 3 AIX 6.1 machines running INFORMIX 11.7 database engine.
One of these servers is the database server and the other 2 servers are connecting to it.
I am doing a test to determine the time of query execution between these servers and i see that in specific times one of these server is taking much more response time.
Here is an example of the test:

Code:

server1: 11:00:00     11:01:01    11:02:00   11:03:00   11:04:00
              0m0,05s     0m0,08s    0m0,08s   0m0,05s   0m0,05s

server2: 11:00:00    11:01:00    11:02:00   11:03:00   11:04:00
              0m0,08s    0m0,10s    0m0,06s   0m0,08s   0m0,08s

server3:  11:00:00   11:01:00  11:02:00    11:03:00    11:04:00
               0m0,07s  0m0,65s   0m0,27s    0m0,25s   0m0,50s

You can see that the execution time on server3 is much longer that the other 2 and i noticed that this happens at the first 5-6 minutes of each hour.
I am trying to figure out what causes this extra delay during these hours but i cannot find anything. I tried to check the crons to see if there is something automatic that runs during these hours that is causing the delay but i dont think my answer is there.

Any help will be appreciated.
Thank you.

Examine the crontabs! The following command greps the cron jobs that are started at the beginning of an hour.

Code:

awk '/^[^#]/ && $2~/^0*[0-4]$/' /var/spool/cron/crontabs/*

What makes you think the answer is not in the crontabs? (The question is meant serious, what haveyou done to come to that conclusion?)

You might want to start with the basic tool for all things performance: vmstat. Use

Code:

vmstat -tw 1 | tee -a /some/log/file

and analyse the log. See if there is any significant difference between the first 5-6 minutes of an hour and the rest of the time. See if there is a difference between the first 5-6 minutes of the hour on server3 and the other two servers.

I hope this helps.

bakunin

Hello guys,

Thank you for your responses.
Well to answer your question about the crons, the crons on server3 are schedules during the early morning hours and from the crons that are running during working hours i did some test.
For the crons that are running at the last or first minutes of an hour i tried to disable one cron at a time to see if there will be any improvement on the time of the execution. When i see that there was no difference then i enabled that cron again and for the next hour i disabled a different one. I ended up disabling all crons one at a time and i get no improvement.
Thats why i said that probably the answer is not there.
Except if you have any suggestion of a different way to test it.
By the way i followed your suggestion to create a

Code:

vmstat

report. I will let it run for few hours and after that i will examine it.

i let vmstat run for 2 hours and i look at the reports to see the output.
the cpu usage of the server never reaches 100% and also it doesnt seem to have any great variation comparing to the time before and after the delay is experienced.

That was not the question. If you want to find out yourself what infos you can glean from a vmstat-output you might want to read a little treatise about the topic.

I hope this helps.

bakunin

thank you bakunin.
the link is indeed very useful!
however when i read the reports the server3 which is having the problem with the delay seems healthy (meaning no paging in/out, no block, wait is low)
one of the other servers shows some block from time to time but that server doesnt have any problem in the aspect of sql delay.