Poor disk performance however no sign of failure


 
Thread Tools Search this Thread
Operating Systems Solaris Poor disk performance however no sign of failure
# 1  
Old 01-04-2012
Poor disk performance however no sign of failure

Hello guys,
I have two servers performing the same disk operations. I believe one server is having a disk's impending failure however I have no hard evidence to prove it. This is a pair of Netra 210's with 2 drives in a hardware raid mirror (LSI raid controller). While performing intensive reads and writes, the system gets backed up with data that previously it was able to handle without issue.

Raidctl -l shows the array is still in optimal condition however iostat shows the wsvc_t and asvc_t being much more excessive in comparision on the server with the potential problem. iostat -Exn only shows 2 soft errors, 0 hard errors, and 0 transport errors.

The load is ranging between 1.25 and 2.0 and cpu utilization is not going above 40%. The server is not heavily using memory at this time either.

What else can I look at to help identify this problem. Thanks for looking.
# 2  
Old 01-04-2012
Plenty to look for here, not that this is an easy answer:-
  • Using vmstat (see your man page for what your output shows you) is your server paging a lot? Consider the placement of the page volumes/files. If you have matching memory and potentially there is a process consuming lost of memory, have a look with something like ps el|sort -n +9 based on the AIX version of ps so you will need to carefully read your man page for that. Take care to check if you want the flags with or without the leading hyphen.
  • Is there a process you don't expect running disk sync all the time? We have users of SQL tools forgetting where they are an initiating /usr/bin/update by mistake and that cripples us sometimes.
  • Are the disks actually comparable?
  • Are you the only user of both servers or is something else skewing your results?
  • Have you recently replaced a disk and one server is still mirroring? Are the RAID controller status displays showing that you are fully operational?
  • Is anything else hitting your network card and causing the server to spend some time responding to that?

Sorry to be soooo vague, but it's one of the less fun things you have to do as the system manager (more than just an administrator) in tracing what's going on and looking for contention. It can prove a costly time investment.



I hope that this helps somewhere, but I'm sure there will be other suggestions to trawl through too.



Robin
Liverpool/Blackburn
UK
# 3  
Old 01-04-2012
Thanks for the suggestions. I have checked most of that with no avail. I found that format has some disk analyzing tools built in however I have questions about whether this will damage data on the system.. These are the options I am interested in. Now, how can one option not harm SunOS and the other two not harm data? Has anyone performed these tests?



format> analyze


ANALYZE MENU:
read - read only test (doesn't harm SunOS)
refresh - read then write (doesn't harm data)
test - pattern testing (doesn't harm data)
# 4  
Old 01-04-2012
Smilie Eeek! Time to be careful. Smilie

On Solaris/SunOS, the format command is the disk slicing tool mainly. You can destroy the system pretty easily with it, as I know to my cost Smilie

The analyze tests you have found shoud be okay to run and they will look for dodgy disk blocks and perhas will flush something out, but they will really hurt performance whilst you run them. I'm not sure if you have to have all the filesystems on those disks unmounted first. It's been such a long time Smilie


Just because I'm paranoid, make sure you get a good backup before you start, then read that manual pages several times to be sure. Smilie




Robin
Liverpool/Blackburn
UK
# 5  
Old 01-06-2012
Thanks Robin,
I was able to run the command against our server in the lab without any problems for a read test. In order to run the other commands you have to have the disk unmounted.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Windows & DOS: Issues & Discussions

Poor Windows 10 Performance of Parallels Desktop 15 on macOS Catalina

Just a quick note for macOS users. I just installed (and removed) Parallels Desktop 15 Edition on my MacPro (2013) with 64GB memory and 12-cores, which is running the latest version of macOS Catalina as of this post. The reason for this install was to test some RIGOL test gear software which... (6 Replies)
Discussion started by: Neo
6 Replies

2. AIX

ISCSI poor performance 1.5MB/s fresh install AIX7.1

Hi Everyone, I have been struggling for few days with iSCSI and thought I could get some help on the forum... fresh install of AIX7.1 TL4 on Power 710, The rootvg relies on 3 SAS disks in RAID 0, 32GB Memory The lpar Profile is using all of the managed system's resources. I have connected... (11 Replies)
Discussion started by: frenchy59
11 Replies

3. Solaris

Poor performance on an M3000

Hi We have an M3000 single physical processor and 8gb of memory running Solaris 10. This system runs two Oracle Databases one on Oracle 9i and One on Oracle 10g. As soon as the Oracle 10g database starts we see an immediate drop in system performance, for example opening an ssh session can... (6 Replies)
Discussion started by: gregsih
6 Replies

4. AIX

Poor Performance of server

Hi, I am new registered user here in this UNIX forums. I am a new system administrator for AIX 6.1. One of our servers performs poorly every time our application (FINACLE) runs many processes/instances. (see below for topas snapshot) I use NMON or Topas to monitor the server utilization. I... (9 Replies)
Discussion started by: guzzelle
9 Replies

5. Solaris

Poor Disk performance on ZFS

Hello, we have a machine with Solaris Express 11, 2 LSI 9211 8i SAS 2 controllers (multipath to disks), multiport backplane, 16 Seagate Cheetah 15K RPM disks. Each disk has a sequential performance of 220/230 MB/s and in fact if I do a dd if=/dev/zero of=/dev/rdsk/<diskID_1> bs=1024k... (1 Reply)
Discussion started by: golemico
1 Replies

6. UNIX for Advanced & Expert Users

HW Raid poor io performance

Hello all We just built a storage cluster for our new xenserver farm. Using 3ware 9650SE raid controllers with 8 x 1TB WD sata disks in a raid 5, 256KB stripe size. While making first performance test on the local storage server using dd (which simulates the read/write access to the disk... (1 Reply)
Discussion started by: roli8200
1 Replies

7. UNIX for Dummies Questions & Answers

poor performance processing file with awk

Hello, I'm running a script on AIX to process lines in a file. I need to enclose the second column in quotation marks and write each line to a new file. I've come up with the following: #!/bin/ksh filename=$1 exec >> $filename.new cat $filename | while read LINE do echo $LINE | awk... (2 Replies)
Discussion started by: scooter53080
2 Replies

8. Filesystems, Disks and Memory

Poor read performance on sun storedge a1000

Hello, i have a a1000 connected to an e6500. There's a raid 10 (12 disks) on the a1000. If i do a dd if=/dev/zero of=/mnt/1 bs=1024k count=1000 and then look at iostat it tells me there's a kw/s of 25000. But if i do a dd of=/dev/zero if=/mnt/1 bs=1024k count=1000 then i see only a... (1 Reply)
Discussion started by: mbrenner
1 Replies

9. UNIX for Advanced & Expert Users

Samba on E3500 Poor Performance!!!

Hi you all, I have a BIG performance problem on an Sun E3500, the scenario is described below: I have several users (30) accessing via samba to the E3500 using an application built on Visual Foxpro from their Windows PC , the problem is that the first guy that logs in demands 30% of the E3500... (2 Replies)
Discussion started by: alex blanco
2 Replies
Login or Register to Ask a Question