08-04-2009
Linux disk performance
I am getting absolutely dreadful iowait stats on my disks when I am trying to install some applications.
I have 2 physical disks on which I have created 2 separate logical volume groups and a logical volume in each. I have dumped some stats as below
My dual core CPU is not being over utilised - 30 to 40% utilisation but the disk i/o wait is in the 70 to 80% range.
Any ideas of what could be degrading disk performance so ?
[root@ebiz1 ~]# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
71609640 50967164 16946248 76% /
/dev/mapper/VolGroup01-LogVol03
721077416 35042564 649406168 6% /u01
/dev/hdc1 101086 11871 83996 13% /boot
tmpfs 1545640 0 1545640 0% /dev/shm
[root@ebiz1 ~]# sar 5 5
Linux 2.6.18-128.el5 (ebiz1.northgate-is.com) 08/04/2009
04:34:08 PM CPU %user %nice %system %iowait %steal %idle
04:34:13 PM all 3.46 0.00 14.17 61.18 0.00 21.2 0
04:34:18 PM all 3.23 0.00 19.40 60.12 0.00 17.2 5
04:34:23 PM all 2.11 0.00 14.08 80.75 0.00 3.0 5
04:34:28 PM all 1.14 0.00 12.31 86.55 0.00 0.0 0
04:34:33 PM all 5.99 0.00 19.98 74.03 0.00 0.00
Average: all 3.14 0.00 15.90 72.60 0.00 8.36
[root@ebiz1 ~]# vmstat -d
disk- ------------reads------------ ------------writes----------- -----IO------
total merged sectors ms total merged sectors ms cur sec
ram0 0 0 0 0 0 0 0 0 0 0
ram1 0 0 0 0 0 0 0 0 0 0
ram2 0 0 0 0 0 0 0 0 0 0
ram3 0 0 0 0 0 0 0 0 0 0
ram4 0 0 0 0 0 0 0 0 0 0
ram5 0 0 0 0 0 0 0 0 0 0
ram6 0 0 0 0 0 0 0 0 0 0
ram7 0 0 0 0 0 0 0 0 0 0
ram8 0 0 0 0 0 0 0 0 0 0
ram9 0 0 0 0 0 0 0 0 0 0
ram10 0 0 0 0 0 0 0 0 0 0
ram11 0 0 0 0 0 0 0 0 0 0
ram12 0 0 0 0 0 0 0 0 0 0
ram13 0 0 0 0 0 0 0 0 0 0
ram14 0 0 0 0 0 0 0 0 0 0
ram15 0 0 0 0 0 0 0 0 0 0
hdc 55278 62611 12423468 14122068 284273 127680 3329180 44306342 0 9170
hdd 22246 4615 999212 777089 86526 6531679 52845920 1606956424 0 12056
dm-0 116332 0 12419066 35397657 416117 0 3328936 339026889 0 9169
dm-1 113 0 904 2199 29 0 232 22845 0 3
hda 0 0 0 0 0 0 0 0 0 0
md0 0 0 0 0 0 0 0 0 0 0
dm-2 25977 0 997514 1013661 6620817 0 52966536 357126262 15 12053
9 More Discussions You Might Find Interesting
1. Filesystems, Disks and Memory
I have some questions regarding disk perfomance, and what I can do to make it just a little (or much :)) more faster.
From what I've heard the first partitions will be faster than the later ones because tracks at the outer edges of a hard drive platter simply moves faster. But I've also read in... (4 Replies)
Discussion started by: J.P
4 Replies
2. AIX
Can I change any AIX System paramerter for speeding the data Disk performance?
Currently it slows with writing operations. (1 Reply)
Discussion started by: gogogo
1 Replies
3. News, Links, Events and Announcements
About 4 years ago I wrote this tool inspired by Rob Urban's collect tool for DEC's Tru64 Unix. What makes this tool as different as collect was in its day is its ability to run at a low overhead and collect tons of stuff. I've expanded the general concept and even include data not available in... (0 Replies)
Discussion started by: MarkSeger
0 Replies
4. AIX
Hello,
I have a aix 570 system with san disk. I do write test of performance
in a lv with four disk. While the test I run filemon tools for trace
the disk activity. The outputs of filemon are at the en of this message. I
see my lV(logical volume) throughput at 100 meg by second. 2 of 4
disk... (0 Replies)
Discussion started by: Hugues
0 Replies
5. Red Hat
Running CentOS 5.5:
I've come across a relatively recent problem, where in the last 2 months or so, the root disk goes to 99% utilization for about 20 seconds when a user logs in. This occurs whether a user logs in locally or via ssh. I have tried using lsof to track down the process that is... (5 Replies)
Discussion started by: dangral
5 Replies
6. Solaris
What tools/utilities do you use to generate metrics on disk i/o throughput on Solaris. For example, if I want to see the i/o rate of random or sequential r/w. (2 Replies)
Discussion started by: dangral
2 Replies
7. Solaris
Hello,
we have a machine with Solaris Express 11, 2 LSI 9211 8i SAS 2 controllers (multipath to disks), multiport backplane, 16 Seagate Cheetah 15K RPM disks.
Each disk has a sequential performance of 220/230 MB/s and in fact if I do a
dd if=/dev/zero of=/dev/rdsk/<diskID_1> bs=1024k... (1 Reply)
Discussion started by: golemico
1 Replies
8. Solaris
Hello guys,
I have two servers performing the same disk operations. I believe one server is having a disk's impending failure however I have no hard evidence to prove it. This is a pair of Netra 210's with 2 drives in a hardware raid mirror (LSI raid controller). While performing intensive... (4 Replies)
Discussion started by: s ladd
4 Replies
9. Linux
I have a freshly installed Oracle Linux 7.1 ( akin to RHEL ) server.
However after installing some Oracle software, I have noticed that my hard disk light is continually on and the system performance is slow.
So I check out SAR and IOSTAT
lab3:/root>iostat
Linux... (2 Replies)
Discussion started by: jimthompson
2 Replies
VMSTAT(8) Linux Administrator's Manual VMSTAT(8)
NAME
vmstat - Report virtual memory statistics
SYNOPSIS
vmstat [-a] [-n] [delay [ count]]
vmstat [-f] [-s] [-m]
vmstat [-S unit]
vmstat [-d]
vmstat [-D]
vmstat [-p disk partition]
vmstat [-V]
DESCRIPTION
vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity.
The first report produced gives averages since the last reboot. Additional reports give information on a sampling period of length delay.
The process and memory reports are instantaneous in either case.
Options
The -a switch displays active/inactive memory, given a 2.5.41 kernel or better.
The -f switch displays the number of forks since boot. This includes the fork, vfork, and clone system calls, and is equivalent to the
total number of tasks created. Each process is represented by one or more tasks, depending on thread usage. This display does not repeat.
The -m displays slabinfo.
The -n switch causes the header to be displayed only once rather than periodically.
The -s switch displays a table of various event counters and memory statistics. This display does not repeat.
delay is the delay between updates in seconds. If no delay is specified, only one report is printed with the average values since boot.
count is the number of updates. If no count is specified and delay is defined, count defaults to infinity.
The -d reports disk statistics (2.5.70 or above required)
The -D reports some summary statistics about disk activity.
The -p followed by some partition name for detailed statistics (2.5.70 or above required)
The -S followed by k or K or m or M switches changes the units of ouput from bytes to outputs between 1000, 1024, 1000000, or 1048576
bytes. Note this does not change the swap (si/so) or block (bi/bo) fields.
The -V switch results in displaying version information.
FIELD DESCRIPTION FOR VM MODE
Procs
r: The number of processes waiting for run time.
b: The number of processes in uninterruptible sleep.
Memory
swpd: the amount of virtual memory used.
free: the amount of idle memory.
buff: the amount of memory used as buffers.
cache: the amount of memory used as cache.
inact: the amount of inactive memory. (-a option)
active: the amount of active memory. (-a option)
Swap
si: Amount of memory swapped in from disk (/s).
so: Amount of memory swapped to disk (/s).
IO
bi: Blocks received from a block device (blocks/s).
bo: Blocks sent to a block device (blocks/s).
System
in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.
CPU
These are percentages of total CPU time.
us: Time spent running non-kernel code. (user time, including nice time)
sy: Time spent running kernel code. (system time)
id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.
st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.
FIELD DESCRIPTION FOR DISK MODE
Reads
total: Total reads completed successfully
merged: grouped reads (resulting in one I/O)
sectors: Sectors read successfully
ms: milliseconds spent reading
Writes
total: Total writes completed successfully
merged: grouped writes (resulting in one I/O)
sectors: Sectors written successfully
ms: milliseconds spent writing
IO
cur: I/O in progress
s: seconds spent for I/O
FIELD DESCRIPTION FOR DISK PARTITION MODE
reads: Total number of reads issued to this partition
read sectors: Total read sectors for partition
writes : Total number of writes issued to this partition
requested writes: Total number of write requests made for partition
FIELD DESCRIPTION FOR SLAB MODE
cache: Cache name
num: Number of currently active objects
total: Total number of available objects
size: Size of each object
pages: Number of pages with at least one active object
NOTES
vmstat does not require special permissions.
These reports are intended to help identify system bottlenecks. Linux vmstat does not count itself as a running process.
All linux blocks are currently 1024 bytes. Old kernels may report blocks as 512 bytes, 2048 bytes, or 4096 bytes.
Since procps 3.1.9, vmstat lets you choose units (k, K, m, M) default is K (1024 bytes) in the default mode
vmstat uses slabinfo 1.1 FIXME
FILES
/proc/meminfo
/proc/stat
/proc/*/stat
SEE ALSO
iostat(1), sar(1), mpstat(1), ps(1), top(1), free(1)
BUGS
Does not tabulate the block io per device or count the number of system calls.
AUTHORS
Written by Henry Ware <al172@yfn.ysu.edu>.
Fabian Frederick <ffrederick@users.sourceforge.net> (diskstat, slab, partitions...)
Throatwobbler Ginkgo Labs 2009 Jan 9 VMSTAT(8)