Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

Process(s) ID waiting on IO?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Process(s) ID waiting on IO?
# 1  
Old 03-12-2007
Process(s) ID waiting on IO?
Hello Experts!!

My CPU is waiting a lot (around 33%) on I/O. I would like to find out what process(s) are waiting on the i/o. Below is my real time output of vmstat and sar.

Thanks for you help !!!!

Regards

Citrus

OS: AIX - 5L

[oracle @ test]: /u2/oracle >oslevel
5.3.0.0
[oracle @ test]: /u2/oracle >oslevel -r
5300-02

*********************************************************
vmstat output
*********************************************************
[oracle @ test]: /u2/oracle >vmstat 2

System configuration: lcpu=32 mem=32768MB

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
3 0 6400821 543969 0 0 0 0 0 0 5896 101632 24882 15 3 49 33
3 1 6400901 544145 0 0 0 331 1079 0 5536 85406 21408 14 3 49 34
3 0 6406515 537957 0 1 0 0 0 0 5846 100449 22468 17 3 47 33
3 0 6408728 536509 0 1 0 403 1176 0 5824 104706 23958 18 3 45 33
5 2 6407629 537001 0 0 0 0 0 0 5851 101328 26252 15 3 48 34


*********************************************************
sar output
*********************************************************

[oracle @ test]: /u2/oracle >sar

10:10:00 4 1 32 63 16.00
10:20:00 4 1 33 62 16.00
10:30:00 4 1 33 63 16.00
10:40:00 5 1 32 62 16.00
10:50:00 6 1 32 61 16.00
11:00:00 10 1 30 58 16.00
11:10:00 17 3 24 56 16.00
11:20:00 17 3 23 58 16.00
11:30:00 16 3 18 63 16.00
11:40:00 15 2 16 67 16.00
11:50:01 15 2 20 63 16.00
12:00:00 15 3 16 66 16.00
12:10:00 14 2 16 67 16.00
12:20:00 15 2 16 67 16.00
12:30:01 15 2 14 69 16.00
12:40:00 15 2 15 67 16.00
Last edited by Citrus143; 03-12-2007 at 06:55 AM..
# 2  
Old 03-12-2007
Buffered I/O over "slow" io ports like a tty mean that the io is waiting on somebody to type something. If you have a lot of interactive users in Ocacle forms this waiting might be considered normal.

It can also mean that you have some really bad queries that result in way more disk I/O per fetch than should happen if they were tuned a bit.

A lot of I/O problems are not kernel tuning issues or load balancing problems, they arise in programmerland and userland....

-- assuming I understood what you mean by waiting.
# 3  
Old 03-12-2007
Quote:
Originally Posted by Citrus143
Hello Experts!!

My CPU is waiting a lot (around 33%) on I/O. I would like to find out what process(s) are waiting on the i/o. Below is my real time output of vmstat and sar.

Thanks for you help !!!!

Regards

Citrus

OS: AIX - 5L

[oracle @ test]: /u2/oracle >oslevel
5.3.0.0
[oracle @ test]: /u2/oracle >oslevel -r
5300-02

*********************************************************
vmstat output
*********************************************************
[oracle @ test]: /u2/oracle >vmstat 2

System configuration: lcpu=32 mem=32768MB

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
3 0 6400821 543969 0 0 0 0 0 0 5896 101632 24882 15 3 49 33
3 1 6400901 544145 0 0 0 331 1079 0 5536 85406 21408 14 3 49 34
3 0 6406515 537957 0 1 0 0 0 0 5846 100449 22468 17 3 47 33
3 0 6408728 536509 0 1 0 403 1176 0 5824 104706 23958 18 3 45 33
5 2 6407629 537001 0 0 0 0 0 0 5851 101328 26252 15 3 48 34


*********************************************************
sar output
*********************************************************

[oracle @ test]: /u2/oracle >sar

10:10:00 4 1 32 63 16.00
10:20:00 4 1 33 62 16.00
10:30:00 4 1 33 63 16.00
10:40:00 5 1 32 62 16.00
10:50:00 6 1 32 61 16.00
11:00:00 10 1 30 58 16.00
11:10:00 17 3 24 56 16.00
11:20:00 17 3 23 58 16.00
11:30:00 16 3 18 63 16.00
11:40:00 15 2 16 67 16.00
11:50:01 15 2 20 63 16.00
12:00:00 15 3 16 66 16.00
12:10:00 14 2 16 67 16.00
12:20:00 15 2 16 67 16.00
12:30:01 15 2 14 69 16.00
12:40:00 15 2 15 67 16.00

Not sure if this link would give you more insight:

http://www-941.ibm.com/collaboration...ype/monitoring
# 4  
Old 03-12-2007
So there isn't a lot of confusion: there is the concept of I/O wait caused by processes being I/O bound; problems with I/O request queue length getting out of hand because resource issues (too few disks); or there is a really "hot" I/O device with a few files that everyone and his mother updates constantly.

I think the point I was and am making is that you have to interpret results in the context of what you know about machine use. I cannot tell from what you posted what is going on without knowing a lot more. The main thing that stands out is "Oracle". Anytime you see that it means that your DBA needs to get involved. Problem Oracle systems root causes are often bad queries and/or lack of tuning.
# 5  
Old 03-13-2007
Thanks a lot for your inputs.

Yes, I agree with Jim that "Oracle systems root causes are often bad queries and/or lack of tuning".

But, here in my case, my database is perfectly good and well tuned. We did not had any Performance Issues on this database for at least 3 months now. Normally, the Idle CPU ranges between 80+ to 98 in our case.

From yesterday at around 5:30PM, the CPU is consistently waiting on I/O (33% wait). From the database, there are no long running queries and only few session are connected to the database and they were mostly idle. It is some OS process which is waiting on I/O and we would like to know which process is it waiting?

Now, for time being lets forget about Oracle, I would like to see which processes are waiting on i/o?

Thanks again for trying to solve the issue!!!
Last edited by Citrus143; 03-13-2007 at 02:36 AM..
# 6  
Old 03-13-2007
Quote:
Originally Posted by Citrus143
Thanks a lot for your inputs.

Yes, I agree with Jim that "Oracle systems root causes are often bad queries and/or lack of tuning".

But, here in my case, my database is perfectly good and well tuned. We did not had any Performance Issues on this database for at least 3 months now. Normally, the Idle CPU ranges between 80+ to 98 in our case.

From yesterday at around 5:30PM, the CPU is consistently waiting on I/O (33% wait). From the database, there are no long running queries and only few session are connected to the database and they were mostly idle. It is some OS process which is waiting on I/O and we would like to know which process is it waiting?

Now, for time being lets forget about Oracle, I would like to see which processes are waiting on i/o?

Thanks again for trying to solve the issue!!!
Yes, thats why I thought maybe that link would help. The vmstat and iostat reports the % or the number of processes blocked on I/O or so. I don't think it gives the process id, so not sure how we can find the process info. If you suspect oracle, then maybe try

ps -o pid,tty,s,args -u <oracle userid>

But don't know if the "s" which denotes the process status really gives good output for process waiting on I/O.
# 7  
Old 03-13-2007
You're right, Deal. The 's' status can mean a lot of other things, including simply sleeping.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Subprocess.popen() should write to log without waiting for process to complete

Here is my test code process = sp.Popen( + , bufsize=1, universal_newlines=True, stdout=sp.PIPE, stderr=sp.STDOUT, cwd=src_home) output, _ =... (2 Replies)
Discussion started by: ezee
2 Replies

2. Shell Programming and Scripting

How to detect that foreground process is waiting for input?

Hi, I have to create a script (ksh or perl) that starts certain number of parallel jobs (another scripts), each of them runs as a foreground process in a separate session. Plus I start monitoring job that has to determine if any of those scripts is expecting input from operator, and switch to... (4 Replies)
Discussion started by: andyh80
4 Replies

3. Shell Programming and Scripting

Waiting for a process to complete in shell script

Hi, I need to initiate a process script which will start and do some processing and then shuts down. Then i need to other verifications. But the the process takes around 25 to 3o minutes. One thing i can monitor the nohup.out file for this process where i can wait for shutting down statement to... (3 Replies)
Discussion started by: Prashanth19
3 Replies

4. Shell Programming and Scripting

Waiting for the background process to finish

Hi, I have on shell script which internally calls more than one scripts which run in background. These scripts cannot be modified to run in foreground. eg. myscript.sh -> bulk_launcher.sh -> acq_launcher.sh -> bulk_loader.sh I want the calling shell script myscript.sh to wait till the... (7 Replies)
Discussion started by: AB10
7 Replies

5. Shell Programming and Scripting

Waiting for an arbitrary background process (limiting number of jobs running)

Hi, I'm trying to write a script to decompress a directory full of files. The decompression commands can run in the background, so that many can run at once. But I want to limit the number running at any one time, so that I don't overload the machine. Something like this: n=0 for i in *.gz... (15 Replies)
Discussion started by: p.f.moore
15 Replies

6. Shell Programming and Scripting

not waiting...why?

My shell script is not waiting. I right click on a file and say convert it to whatever and it runs this script. It converts it using Compressor but I want it to wait until it is 100% done before moving on and it is not waiting. I have tried to put it in the background and using "wait". I have tried... (10 Replies)
Discussion started by: mainegate
10 Replies

7. Shell Programming and Scripting

not waiting

I have a script that runs Compressor and converts the file then I want it to inject the file with Flvtool2. My script works fine but the flvtool2 is starting too early. I have tried to put it in the background by putting a "&" sign at the end and then put a "wait" on a new line. Anyone have any... (2 Replies)
Discussion started by: mainegate
2 Replies

8. Shell Programming and Scripting

Waiting for a background process to finish

Hello, I am a novice shell script programmer. And facing this problem any help is appreciated. I m writing a shell script and running few commands in it background as I have to run them simultaneously. Sample code : sql_prog & sql_prog & sql_prog & echo "Process Completed" Here... (2 Replies)
Discussion started by: vineetbhati
2 Replies

9. AIX

Waiting time of Process

Hi all, I am trying to find out the process wait time on Unix(AIX/SOLARIS) machine( only sh/ksh/csh): Like EXAMPLE 1 : $ vmstat 2 System configuration: lcpu=16 mem=32000MB kthr memory page faults cpu ----- -----------... (4 Replies)
Discussion started by: chandrakala.sg
4 Replies

10. Linux

waiting process

how to know the information of the waiting process how to calculate the time of the process that it has taken to execute i want to make a program that Should be able to keep a log of the processes expired(The log should contain the starting time, expiry time, time slices used, total execution... (2 Replies)
Discussion started by: shukla_chanchal
2 Replies
Login or Register to Ask a Question