Disk IO high. What to fix ?


 
Thread Tools Search this Thread
Operating Systems Solaris Disk IO high. What to fix ?
# 1  
Old 09-30-2013
Disk IO high. What to fix ?

This is Solaris-10 box. I can see disk I/O is high and performance is very very slow for applications running on it. What should I check to fix this issue ?
Code:
root@dbrpd01:/# iostat -xntz | head -7
   tty
 tin tout
   7   79
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   40.1   80.2  376.4  605.1  2.1  4.3   17.7   35.9   1  10 c1t0d0
   31.1   69.6  290.2  534.4  1.6  3.1   16.2   30.9   1   7 c1t1d0
root@dbrpd01:/# vmstat 5 5
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s1 s2 s3 sd   in   sy   cs us sy id
 18 0 0 71728216 21073504 1401 17657 4357 5 5 0 0 120 101 -0 -7505 14249 182183 23659 13 12 76
 23 30 0 95227096 37593168 73 5350 4187 2 2 0 0 386 154 0 0 11169 393110 17954 14 6 80
 2 30 0 95187504 37529632 302 5818 5619 2 2 0 0 403 299 0 12 11549 278131 18816 14 7 80
 13 31 0 95142896 37477960 142 5094 6013 0 0 0 0 381 357 0 4 13089 415174 37007 12 8 79
 9 33 0 95144816 37460264 760 13224 4736 0 0 0 0 394 204 0 0 12028 302935 31685 12 9 80
root@dbrpd01:/#
root@dbrpd01:/#
root@dbrpd01:/# sar -b
SunOS dbrpd01 5.10 Generic_144488-17 sun4u    09/30/2013
00:00:01 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
00:20:01       6    1459     100      36     186      81      70       0
00:40:01       4     666      99      33     139      76      62       0
01:00:01      21    1397      99      81     477      83      71       0
Average       10    1174      99      50     267      81      68       0
root@dbrpd01:/#
root@dbrpd01:/# iostat -En | grep Hard | head -2
c1t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
c1t1d0           Soft Errors: 0 Hard Errors: 2 Transport Errors: 2
root@dbrpd01:/#

While I do not see any major hardware errors in iostat. Its uptime is 190 days.
Code:
/# ps -ef | grep -i defunct | wc -l
      18

Above process are also not large in numbers, so it may not be contributing to slowness of applications.
# 2  
Old 09-30-2013
defunct processes seldom contribute to high IO wait times anyway. (Although they do suggest something is wonky, even Solaris doesn't usually have 18 of those things floating around after such a short uptime (half a year isn't long).

First identify if it's a tuning problem, a rubbish IO subsystem, or a rubbish application.
Is it so slow when the applications are not running? (ie bare OS, nothing else in memory)
Is it slow if you dd a few gigs of /dev/urandom out to a temp file on the disk?

If it's slow even without the app, see if the performance varies between the two disks, look for faults on the slower one if they differ a lot. Look for something wrong with the interface card if it's both the same.
If it's slow when the app runs, try to identify what process is spending all it's time in iowait and start there.
If these are not local disks (ie SAN perhaps), check your block sizes are sensible, exact multiples of each other, check for excessive contention on the fabric switch, yell at your SAN admin some - it's always their fault Smilie
# 3  
Old 10-10-2013
Do you have a ORACLE db running on this server? sometimes a badly written SQL query can cause the slowdown especially if you have the table spaces configured on slower internal disks.
# 4  
Old 10-22-2013
Are these your root disk? If so what volume management are you using?
# 5  
Old 10-23-2013
Heavy Swapping Perhaps?

try doing
Code:
vmstat 4


output:

Code:
kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr vc vc -- --   in   sy   cs us sy id
 0 0 0 10794904 3118992 96 1023 193 2 2 0 0 10 0  0  0  555 1430  556  2  5 93
 0 0 0 10717432 3083328 0 3 22  0  0  0  0  0  0  0  0  452  165  399  0  1 99

if any of the first two columns are incrementing you are running out of memory and writing to swap.

Last edited by os2mac; 10-23-2013 at 04:34 PM.. Reason: spelling...
# 6  
Old 10-23-2013
I was able to fix it with a reboot. Reboot was scheduled for a different issue and post reboot, I do not see this issue anymore. Thanks for your inputs
# 7  
Old 10-23-2013
Quote:
Originally Posted by os2mac

Code:
kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr vc vc -- --   in   sy   cs us sy id
 0 0 0 10794904 3118992 96 1023 193 2 2 0 0 10 0  0  0  555 1430  556  2  5 93
 0 0 0 10717432 3083328 0 3 22  0  0  0  0  0  0  0  0  452  165  399  0  1 99

if any of the first two columns are incrementing you are running out of memory and writing to swap.
Not really. The first two columns are the run and blocked thread queue, nothing related to swap issues. Better to monitor the third (wait) and twelfth (scan rate) ones if you suspect this.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Emergency UNIX and Linux Support

How to find which process using, while getting High Disk I/O?

In Our Production server I/O was very high, I Recived mail that Disk I/O was high, is it possible how to find which process Used this much I/O ? Iam Using Ubuntu server 12.04. Linux 3.9.3-x86_64-server33 (Li473-1200) 07/23/2014 _x86_64_ (8 CPU) 12:05:01 AM DEV ... (5 Replies)
Discussion started by: babinlonston
5 Replies

2. Shell Programming and Scripting

what would a script include to find CPU's %system time high and user time high?

Hi , I am trying to :wall: my head while scripting ..I am really new to this stuff , never did it before :( . how to find cpu's system high time and user time high in a script?? thanks , help would be appreciated ! :) (9 Replies)
Discussion started by: sushwey
9 Replies

3. Red Hat

apache high cpu load on high traffic

i have a Intel Quad Core Xeon X3440 (4 x 2.53GHz, 8MB Cache, Hyper Threaded) with 16gig and 1tb harddrive with a 1gb port and my apache is causing my cpu to go up to 100% on all four cores heres my http.config <IfModule prefork.c> StartServers 10 MinSpareServers 10 MaxSpareServers 15... (4 Replies)
Discussion started by: awww
4 Replies

4. AIX

Diagnose high disk write IO

Hi, say for example if there is high disk write IO in one disk (detected from NMON), how to we identify what processes is writing on that particular disk? (3 Replies)
Discussion started by: ngaisteve1
3 Replies

5. Red Hat

Disk Utilization is very high

Hi, I have monitored that disk utilization is very high on one of red hat linux VM. Would like to know how to find out that issue of high disk utilization is because of disk or Installed Application on that server is causing the problem. Regards, Manoj (1 Reply)
Discussion started by: manoj.solaris
1 Replies

6. UNIX for Advanced & Expert Users

identify the unix process performing high disk i/o reads and writes

Guys, Is there any UNIX command that captures the 'Unix process which is performing high disk I/O reads and writes'. can you help me in this? -Swamy (6 Replies)
Discussion started by: avsswamy
6 Replies

7. Shell Programming and Scripting

identify the unix processes performing high disk i/o reads and writes

I would like to write shell/perl script which identifies the top unix processes that are performing high disk I/O's or/and writes If any one knows the solution please help me? -Swamy (0 Replies)
Discussion started by: avsswamy
0 Replies

8. UNIX for Dummies Questions & Answers

How to trace the big files causing high disk usage

Hi All, One very urgent issue and I need your help. I have two V490 servers installed with Sun Solaris 10. I have traced out that the disk usage is running out of space for both the servers. Currently the /dev/md/dsk/d10 device is 91% in one server and another is 56% and it is increasing... (1 Reply)
Discussion started by: ailnilanjan
1 Replies

9. Shell Programming and Scripting

Need to send email on HIGH Disk usage

Hi Guys I am looking for a python / PERL script which will send me email when ever my disk becomes more than 90% full. By the way my OS is Win XP. If anybody have already has written same type of script or something very similar kind of script, that will also be very helpful. Thanks... (1 Reply)
Discussion started by: csaha
1 Replies

10. UNIX for Advanced & Expert Users

Sun: High kernel usage & very high load averages

Hi, I am seeing very high kernel usage and very high load averages on my system (Although we are not loading much data to our database). Here is the output of top...does anyone know what i should be looking at? Thanks, Lorraine last pid: 13144; load averages: 22.32, 19.81, 16.78 ... (4 Replies)
Discussion started by: lorrainenineill
4 Replies
Login or Register to Ask a Question