Gurus, i have process that runs 5 times a day.
it runs normally (takes about 1 hour) to complete in 3 runs
but it is takes about ( 3 hrs to complete) two times
So i need to figure out why it takes significanlty high time during
those 2 runs.
The process is a shell script that connect to database and fetches some data
through simple select statement(in a stored procedure)
i ran a iostat 10 in the machine during normal and extented time
and compared the cpu utilisation and checks to see if there is any other
process that is taking the bulk of cpu usage but didn't find
any.
The only differnce was on cpu utilization where the process took .3 % of cpu
during normal runs and .1% during extended run
normal run
Code:
root 516 44.2 1.0 8 6012 - A Apr 22 8500:53 kproc
root 19618 15.2 1.0 10404 10148 - A 22:30:50 2:34 sendmail: DAF0424
root 23072 14.3 3.0 24428 24176 - A 22:00:50 6:43 sendmail: DBX2008
sybase 13166 9.1 0.0 4060 2452 - A Apr 22 1742:49 /sybase/syb1103/b
root 11098 2.6 6.0 88020 58316 - A Apr 22 509:52 /usr/bin/dsmc sch
root 3396 0.4 0.0 156 36 - A Apr 22 78:52 /usr/sbin/syncd 6
sa 16098 0.4 0.0 1876 1420 - A 20:10:55 0:37 sshd: sa@pts/1
root 1032 0.2 1.0 64 6060 - A Apr 22 47:31 kproc
sa 18556 0.2 0.0 720 724 - A 22:26:57 0:03 isql -P
The last process isql -P being my process
Please advice on how to find the bottleneck.
PS: vmstat doesn't work in it ,My /var is about 90% does that have any effect in it?
---------- Post updated at 02:05 PM ---------- Previous update was at 01:48 PM ----------
You could check several things:
Make sure your box has no general performance problem. This can be achived with running topas or vmstat (get it working! ) instead of just iostat since they give much more interessting info on virtual memory (paging space in/outs etc.), kernel thread queues etc...
Make sure there is no other jobs running at this time in parallel which were not running in the days before when everything was fine.
Set up nmon and nmontools (nmon2rrd) for long term monitoring (free available via IBM websites). With the tools you can generate nice looking reports in form of html sites with good formatted graphics so it will be easier to identify in a short time, if there were any obscurities in the different runs, like other processes running in parallel which are not supposed to be there and so on.
Having var at 90% should not be an issue. No idea what isql does but from googling I guess it's just a SQL java client to issue some SQL on a box. Is the database in that case running on the same box or a remote box? You might have to check if the issued runs are different in execution etc. ie. if table scans occure that because of a now missing index or something like that. Need a bit more info about your infrastructure, what you are doing with isql etc. and which app is on which box etc.
The 0.1% to 0.3% should be really no issue. If your box has no room CPU wise for 0.2%, then there you have another big general problem.
Hi,
We have 2 lpars on p6 blade. One of the lpar is having 3 core cpu with 5gb memory running sybase as database. An EOD process takes 25 min. to complete.
Now we have an lpar on P7 server with entitled cpu capacity of 2 with 16 Gb memory and sybase as database. The EOD process which takes... (17 Replies)
hi I am having a performance issue with the following requirement
i have to create a permutation and combination on a set of three files
such that each record in each file is picked and the output is redirected in
a specific format but it is taking around 70 odd hours to prepare a
combination... (7 Replies)
Hi Guys,
This is the situation I am in. Provide your views and input where should I start?
I have one P7 test server and a p520 production server. the job is taking pretty long on the P7 test server when compared to the P5 production server. below is the full detail.
Informix... (5 Replies)
Hi All,
I have the following script which I use in Nagios to check the health of the applications, the problem with it is that the curl part ($TOTAL) does not return anything after running for 2-3 hrs, even though from command line the script runs fine but not from Nagios.
There are 17... (1 Reply)
In my C program i am using very large file(approx 400MB) to read parts of it frequently. But due to large file the performance of the program goes down very badly. It shows very high I/O usage and I/O wait time.
My question is, What are the ways to optimize or tune I/O on linux or how i can get... (10 Replies)
Hi All,
I am getting a performance issue in expect5.43.0 library on IBM AIX 5.3. When I have used exp_fexpectv call for pattern matching with the expect string, the execution latency of the call is approximately 200 ms. In that way, I am able to complete only 4 or 5 transactions per... (2 Replies)
Hi,
on a linux server I have the following :
vmstat 2 10
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 4 0 675236 39836 206060 1617660 3 3 3 6 8 7 1 1 ... (1 Reply)
We have a AIX v5.3 on a p5 system with a poor performing Ingres database.
We added one CPU to the system to see if this would help. Now there are two CPU's.
with sar and topas -P I see good results: CPU usage around 30%
with topas I only see good results in the process output screen, the... (1 Reply)
05-07-2010
) instead of just iostat since they give much more interessting info on virtual memory (paging space in/outs etc.), kernel thread queues etc...