Login or Register to Ask a Question and Join Our Community


Solaris

100% wait and heavy wsvc_t for new harddrive

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Solaris 100% wait and heavy wsvc_t for new harddrive
# 1  
Old 12-01-2009
100% wait and heavy wsvc_t for new harddrive
Hi,

recently we had an hdd failing in my SVM mirror, throwing hard errors; engineer on site swapped the hdd (he did it all right, from as I see - he broke the mirror, unconfigured the hdd, replaced it with the new one, configured and attached as metadevice). But, after this it began to throw i/o wait errors, stating that it constantly in 100% wait, and sometimes throws 2000 transactions in wait queue!

Code:
root@server# iostat -zxnM 1 44
                    extended device statistics              
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.3    1.3    0.0    0.0  0.0  0.0    3.8   15.3   1   1 d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.1   21.2   0   0 d1
    0.0    1.0    0.0    0.0  0.0  0.0    4.0   13.0   0   1 d3
    0.0    0.0    0.0    0.0  0.0  0.0    0.0   10.1   0   0 d4
    0.1    0.4    0.0    0.0  0.0  0.0    1.8    9.6   0   0 d5
    0.2    1.2    0.0    0.0  0.0  0.0    0.0   14.5   0   1 d10
    0.0    0.0    0.0    0.0  0.0  0.0    0.0   28.4   0   0 d11
    0.0    1.0    0.0    0.0  0.0  0.0    0.0   11.6   0   0 d13
    0.0    0.0    0.0    0.0  0.0  0.0    0.0   17.7   0   0 d14
    0.1    0.4    0.0    0.0  0.0  0.0    0.0    8.0   0   0 d15
    0.2    1.2    0.0    0.0  0.0  0.0    0.0   14.2   0   1 d20
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    3.5   0   0 d21
    0.0    1.0    0.0    0.0  0.0  0.0    0.0   11.5   0   1 d23
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    2.6   0   0 d24
    0.1    0.4    0.0    0.0  0.0  0.0    0.0    7.9   0   0 d25
    0.2    3.3    0.0    0.0  0.0  0.0    6.1   13.1   1   2 c1t0d0
    0.2    3.4    0.0    0.0  0.0  0.0    0.0   12.9   0   2 c1t1d0
                     extended device statistics              
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  2.0  0.0    0.0    0.0 100   0 c1t0d0
                    extended device statistics              
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  2.0  0.0    0.0    0.0 100   0 c1t0d0
    0.0    6.0    0.0    0.0  0.0  0.1    0.0   10.3   0   4 c1t1d0
                    extended device statistics              
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    1.0    0.0    0.0  0.0  0.0    0.0   14.1   0   1 d3
    0.0    1.0    0.0    0.0  0.0  0.0    0.0    6.2   0   1 d13
    0.0    1.0    0.0    0.0  0.0  0.0    0.0   14.0   0   1 d23
    0.0    1.0    0.0    0.0  2.0  0.0 2000.0    6.1 100   1 c1t0d0
    0.0    1.0    0.0    0.0  0.0  0.0    0.0   14.0   0   1 c1t1d0

Code:
root@server# metastat d0
d0: Mirror
    Submirror 0: d10
      State: Okay         
    Submirror 1: d20
      State: Okay         
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 30721344 blocks (14 GB)

d10: Submirror of d0
    State: Okay         
    Size: 30721344 blocks (14 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t0d0s0          0     No            Okay   Yes 


d20: Submirror of d0
    State: Okay         
    Size: 30721344 blocks (14 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t1d0s0          0     No            Okay   Yes 


Device Relocation Information:
Device   Reloc  Device ID
c1t0d0   Yes    id1,sd@SFUJITSU_MAW3073NCSUN72G_000701B0J9TK____DAN0P710J9TK
c1t1d0   Yes    id1,sd@SFUJITSU_MAT3073N_SUN72G_000516B03LPL____AAN0P5403LPL

Code:
root@server# df -h
Filesystem             size   used  avail capacity  Mounted on
/dev/md/dsk/d0          14G   3.2G    11G    23%    /
/proc                    0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
/dev/md/dsk/d3         3.9G   1.3G   2.6G    33%    /var
swap                   7.5G    48K   7.5G     1%    /var/run
swap                   9.2G   1.7G   7.5G    19%    /tmp
fd                       0K     0K     0K     0%    /dev/fd
/dev/md/dsk/d5          42G   2.8G    39G     7%    /apps

Code:
root@server# prtdiag 
System Configuration: Sun Microsystems  sun4u Sun Fire V240
System clock frequency: 167 MHZ
Memory size: 4GB

Code:
root@server# uname -a
SunOS chiscweb03 5.9 Generic_118558-06 sun4u sparc SUNW,Sun-Fire-V240

Any ideas on what`s the issue? What could be done? Maybe an good idea to do all the procedure of detatching/attaching & configure/unconfigure again? Or, maybe it`s an controller issue?

Thanks for your ideas.

Nik.
# 2  
Old 12-01-2009
Update your kernel patch on the system and reboot
# 3  
Old 12-02-2009
Quote:
Originally Posted by masloff
But, after this it began to throw i/o wait errors stating that it constantly in 100% wait, and sometimes throws 2000 transactions in wait queue!
Your log actually shows 2 transactions in the wait queue and 2 seconds (2000 ms) of waiting time. Still unexpected values but you should measure what happens under a real load.
Last edited by jlliagre; 12-02-2009 at 06:03 AM..
# 4  
Old 12-02-2009
2incredible - thanks, this is in my plans

2jilliagre - thanks, but what worries me is 100 in %w column, it shows that actually hdd is waiting all the time...
Code:
 r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
 0.0    1.0    0.0    0.0  2.0  0.0 2000.0    6.1 100   1 c1t0d0

# 5  
Old 12-02-2009
You are right, it is waiting 2 seconds to achieve a single write operation and this is certainly an unexpected behavior.
# 6  
Old 12-02-2009
i believe %w > 5 shows some serious io issues. basically the disk is way way too busy. you should figure out what process is writing to disk and see if you can adjust it. maybe a log file? seems odd that you would see this to the root disk. or maybe thats just me Smilie
# 7  
Old 12-03-2009
How can the disk be "too busy" with an activity close to nil ?
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

calling a shell script in background and wait using "wait" in while loop

Hi, I am facing a strange issue, when i call a script from my while loop in background it doesnt go in background, despite the wait i put below the whil loop it goes forward even before the process put in background is completed. cat abc.txt | while read -u4 line do #if line contains #... (2 Replies)
Discussion started by: mihirvora16
2 Replies

2. Red Hat

NFS to HEAVY TRAFFIC

dear all, i need case with my nfs .... Nov 11 15:49:13 BANK02 kernel: nfs: server 10.16.108.5 not responding, still trying Nov 11 15:49:16 BANK02 last message repeated 2 times Nov 11 15:49:20 BANK02 kernel: nfs: server 10.16.108.5 OK Nov 11 15:49:20 BANK02 last message repeated 2 times Nov... (2 Replies)
Discussion started by: zvtral
2 Replies

3. UNIX for Dummies Questions & Answers

How to copy MBR from old harddrive to new harddrive?

How do I copy the master boot record from one harddrive to another or how to install just the MBR? (7 Replies)
Discussion started by: shorty
7 Replies

4. Shell Programming and Scripting

wait command - cat it wait for not-chile process?

Did not use 'wait' yet. How I understand by now the wait works only for child processes, started background. Is there any other way to watch completion of any, not related process (at least, a process, owned by the same user?) I need to start a background process, witch will be waiting... (2 Replies)
Discussion started by: alex_5161
2 Replies

5. UNIX for Dummies Questions & Answers

sed command heavy!!

Is the Sed command heavy on system resources!! Thanks!! (2 Replies)
Discussion started by: aixjadoo
2 Replies

6. UNIX for Dummies Questions & Answers

booting from which harddrive

Hi, both of my /dev/hda and my /dev/hdb contain /boot partition. I'm wondering how to tell which harddrive's /boot is actually being read? Thanks (2 Replies)
Discussion started by: onthetopo
2 Replies

7. Shell Programming and Scripting

Need to execute 2 scripts, wait, execute 2 more wait, till end of file

:cool: I need to execute a shell script to do the following: cat a file run two back ground processes using the first two values from the file wait till those background processes finish run two more background processes using the next two values from the file wait till those background... (1 Reply)
Discussion started by: halo98
1 Replies

8. Filesystems, Disks and Memory

Supplemental harddrive.

Am planning on adding a secodary SCSI hardrive to the existing 20gb drive., that I have. The old drive has Linux on it. Once, the new drive is added, I am planning on having windows on it. Firstly, could this be done ? Has anyone build a system with a similar configuration ? What is requried,... (1 Reply)
Discussion started by: matvrix
1 Replies
Login or Register to Ask a Question