Red Hat

I am getting absolutely dreadful iowait stats on my disks when I am trying to install some applications.

I have 2 physical disks on which I have created 2 separate logical volume groups and a logical volume in each. I have dumped some stats as below

My dual core CPU is not being over utilised - 30 to 40% utilisation but the disk i/o wait is in the 70 to 80% range.

Any ideas of what could be degrading disk performance so ?

[root@ebiz1 ~]# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
71609640 50967164 16946248 76% /
/dev/mapper/VolGroup01-LogVol03
721077416 35042564 649406168 6% /u01
/dev/hdc1 101086 11871 83996 13% /boot
tmpfs 1545640 0 1545640 0% /dev/shm



[root@ebiz1 ~]# sar 5 5

Linux 2.6.18-128.el5 (ebiz1.northgate-is.com) 08/04/2009

04:34:08 PM CPU %user %nice %system %iowait %steal %idle

04:34:13 PM all 3.46 0.00 14.17 61.18 0.00 21.2 0

04:34:18 PM all 3.23 0.00 19.40 60.12 0.00 17.2 5

04:34:23 PM all 2.11 0.00 14.08 80.75 0.00 3.0 5

04:34:28 PM all 1.14 0.00 12.31 86.55 0.00 0.0 0

04:34:33 PM all 5.99 0.00 19.98 74.03 0.00 0.00

Average: all 3.14 0.00 15.90 72.60 0.00 8.36

[root@ebiz1 ~]# vmstat -d
disk- ------------reads------------ ------------writes----------- -----IO------
total merged sectors ms total merged sectors ms cur sec
ram0 0 0 0 0 0 0 0 0 0 0
ram1 0 0 0 0 0 0 0 0 0 0
ram2 0 0 0 0 0 0 0 0 0 0
ram3 0 0 0 0 0 0 0 0 0 0
ram4 0 0 0 0 0 0 0 0 0 0
ram5 0 0 0 0 0 0 0 0 0 0
ram6 0 0 0 0 0 0 0 0 0 0
ram7 0 0 0 0 0 0 0 0 0 0
ram8 0 0 0 0 0 0 0 0 0 0
ram9 0 0 0 0 0 0 0 0 0 0
ram10 0 0 0 0 0 0 0 0 0 0
ram11 0 0 0 0 0 0 0 0 0 0
ram12 0 0 0 0 0 0 0 0 0 0
ram13 0 0 0 0 0 0 0 0 0 0
ram14 0 0 0 0 0 0 0 0 0 0
ram15 0 0 0 0 0 0 0 0 0 0
hdc 55278 62611 12423468 14122068 284273 127680 3329180 44306342 0 9170
hdd 22246 4615 999212 777089 86526 6531679 52845920 1606956424 0 12056
dm-0 116332 0 12419066 35397657 416117 0 3328936 339026889 0 9169
dm-1 113 0 904 2199 29 0 232 22845 0 3
hda 0 0 0 0 0 0 0 0 0 0
md0 0 0 0 0 0 0 0 0 0 0
dm-2 25977 0 997514 1013661 6620817 0 52966536 357126262 15 12053

If you edit your posting and put code tags around them it will preserve the columns.

The obvious way of improving the performance is to make volumes striped over the two disks assuming you can trash the current volumes and rebuild them, take a look at: LVM HOWTO

As one of the two disks you mention has the operating system on it including swap (presumably), put the applications only on the other disk so that they are not competing with the OS for disk bandwidth.

Another consideration is the performance of the disk controller itself. If the controller or the driver for it is poor, your overall will be as well. You also should consider what kernel scheduler to use for your workload as well as the speed of the drives themselves and whether the controller or disks can keep up with your workload type.

What file system were you using? What parameters at mount?

It's not clear what applications are running on this server, but if a HDD or controller fault is suspected, you may install smart monitor tools to check for hardware related issues.