UNIX for Advanced & Expert Users

Hi

I am trying to investigate a disk performance issue, and we are not seem to be hitting the right direction in our analysis.

This is a FC disk running on USP1000 HDS system. The application is an IO intensive application, but our opinion is that it is not performing due to perceived disk issues.

Analysis:

iostat on the disk shows over various iterations, something like:

Disks: %tm_act Kbps tps Kb_read Kb_wrtn time
hdisk40 27 257.5 50.1 2551 12900 0:01:12
hdisk40 7.8 305.6 20.8 9088 9244 0:02:12
hdisk40 0.9 14 2.1 160 680 0:03:12
hdisk40 0.7 8.4 1.3 16 488 0:04:12
hdisk40 2.6 205 9.8 11012 1288 0:05:12
hdisk40 8.9 1055 50.1 57140 6160 0:06:12
hdisk40 6.1 1117.4 49.1 63432 3608 0:07:12
hdisk40 5.6 810.9 35.1 46488 2164 0:08:12
hdisk40 0.7 6.3 1 0 380 0:09:12
hdisk40 7.3 320 12.9 4988 14208 0:10:12
hdisk40 29.4 1751.5 83.3 19948 85128 0:11:12

To calculate the throughput here, is it OK to just take (Kb_read+Kb_wrtn)/1024 to get the throughput in MBPS?

Is iostat a reliable way to calculate throughput or do I have to necessarily use nstress?

Is there a problem with this disk?

Thanks
Theerthan

Hi, I have posted some iostat output (down below) taken at 1 minute intervals during a peak period. We couldn't really conclude whether disk IO is good or bad. The application team claims disk IO to be the cause of the issues. I also see foll. values in vmstat -v, especially:

===
4216 pending disk I/Os blocked with no pbuf
3099 paging space I/Os blocked with no psbuf
28345750 filesystem I/Os blocked with no fsbuf
===

The disk queue depth was set to 2, and numfsbufs was set to 186. Upon seeing these vmstat values, we changed the queue depth to 12 and the numfsbufs to 2048.

After this, the system got even worse. An IO archive operation which took 3 hrs earlier took 8 hours to complete. The system suffered a lot of paging activity, thus the degradation was understood, but the system has 4GB memory and enough paging space. So we reversed the parameters back and we are back to the original performance.

Any directions to move forward will be much appreciated.

rgds
theerthan.

===vmstat -v====
20080125 2024
2097152 memory pages
1991557 lruable pages
11928 free pages
1 memory pools
181487 pinned pages
80.0 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
80.1 numperm percentage
1595588 file pages
0.0 compressed percentage
0 compressed pages
0.0 numclient percentage
80.0 maxclient percentage
0 client pages
0 remote pageouts scheduled
4216 pending disk I/Os blocked with no pbuf
3099 paging space I/Os blocked with no psbuf
28345750 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no fsbuf

====iostat @ 1 minute intervals====

tty: tin tout avg-cpu: % user % sys % idle % iowait time
0.0 0.0 9.9 5.8 80.3 4.1 08:58:13

Disks: % tm_act Kbps tps Kb_read Kb_wrtn time
hdisk1 1.6 32.8 4.1 40 1929 08:58:13
hdisk0 1.9 38.2 5.4 361 1929 08:58:13
hdisk49 0.5 8.1 1.2 8 476 08:58:13
hdisk51 0.0 0.0 0.0 0 0 08:58:13
hdisk47 2.4 228.0 18.7 13408 272 08:58:13
hdisk43 0.2 2.1 0.5 0 128 08:58:13
hdisk53 0.0 0.0 0.0 0 0 08:58:13
hdisk46 0.4 9.1 1.0 536 8 08:58:13
hdisk42 0.0 0.0 0.0 0 0 08:58:13
hdisk39 1.9 30.2 5.5 708 1104 08:58:13
hdisk38 2.0 230.5 18.9 13564 264 08:58:13
hdisk50 0.1 1.2 0.3 4 68 08:58:13
hdisk45 0.0 0.0 0.0 0 0 08:58:13
hdisk44 0.1 0.3 0.1 0 20 08:58:13
hdisk40 0.7 6.7 1.3 0 404 08:58:13
hdisk41 0.1 1.5 0.3 24 64 08:58:13
hdisk52 0.2 1.6 0.3 0 96 08:58:13
hdisk36 0.1 0.0 0.0 0 0 08:58:13
hdisk48 1.9 31.4 5.4 784 1100 08:58:13
hdisk37 0.4 6.6 0.9 396 0 08:58:13
hdisk65 2.1 230.3 19.0 13580 236 08:58:13
hdisk54 0.0 0.0 0.0 0 0 08:58:13
hdisk56 2.2 223.8 18.9 13152 276 08:58:13
hdisk60 0.0 0.0 0.0 0 0 08:58:13
hdisk71 0.0 0.0 0.0 0 0 08:58:13
hdisk64 0.2 6.3 0.7 380 0 08:58:13
hdisk55 0.1 8.9 0.7 536 0 08:58:13
hdisk59 0.0 0.0 0.0 0 0 08:58:13
hdisk62 0.0 0.0 0.0 0 0 08:58:13
hdisk69 0.0 0.0 0.0 0 0 08:58:13
hdisk63 0.0 0.0 0.0 0 0 08:58:13
hdisk66 2.1 25.4 5.7 408 1116 08:58:13
hdisk67 0.6 8.1 1.3 12 472 08:58:13
hdisk58 0.6 6.1 1.1 4 360 08:58:13
hdisk57 2.5 25.2 5.8 460 1052 08:58:13
hdisk68 0.0 0.1 0.0 4 0 08:58:13
hdisk61 0.2 1.9 0.4 0 112 08:58:13
hdisk70 0.3 3.5 0.7 0 208 08:58:13
cd1 0.0 0.0 0.0 0 0 08:58:13
cd0 0.0 0.0 0.0 0 0 08:58:13

tty: tin tout avg-cpu: % user % sys % idle % iowait time
0.0 0.0 10.7 11.1 74.2 4.1 08:59:13

Disks: % tm_act Kbps tps Kb_read Kb_wrtn time
hdisk1 3.6 166.5 9.1 6783 3208 08:59:13
hdisk0 3.3 61.0 8.2 453 3208 08:59:13
hdisk49 1.1 12.1 2.2 0 728 08:59:13
hdisk51 0.0 0.0 0.0 0 0 08:59:13
hdisk47 0.3 1.7 0.4 0 104 08:59:13
hdisk43 0.1 1.0 0.2 0 60 08:59:13
hdisk53 0.2 1.3 0.3 4 76 08:59:13
hdisk46 0.0 0.0 0.0 0 0 08:59:13
hdisk42 0.0 0.0 0.0 0 0 08:59:13
hdisk39 5.3 50.3 11.7 780 2236 08:59:13
hdisk38 0.3 1.5 0.4 0 92 08:59:13
hdisk50 0.1 0.2 0.1 0 12 08:59:13
hdisk45 0.0 0.0 0.0 0 0 08:59:13
hdisk44 0.0 0.3 0.1 0 20 08:59:13
hdisk40 1.3 16.2 2.4 4 968 08:59:13
hdisk41 0.0 0.0 0.0 0 0 08:59:13
hdisk52 0.3 1.7 0.3 0 104 08:59:13
hdisk36 0.1 0.0 0.0 0 0 08:59:13
hdisk48 5.0 50.7 11.5 828 2212 08:59:13
hdisk37 0.0 0.0 0.0 0 0 08:59:13
hdisk65 0.3 1.3 0.3 0 80 08:59:13
hdisk54 0.0 0.0 0.0 0 0 08:59:13
hdisk56 0.2 1.3 0.3 0 80 08:59:13
hdisk60 0.0 0.0 0.0 0 0 08:59:13
hdisk71 0.1 0.7 0.1 0 40 08:59:13
hdisk64 0.0 0.0 0.0 0 0 08:59:13
hdisk55 0.0 0.1 0.0 0 4 08:59:13
hdisk59 0.2 1.5 0.3 0 92 08:59:13
hdisk62 0.1 1.7 0.3 0 100 08:59:13
hdisk69 0.0 0.0 0.0 0 0 08:59:13
hdisk63 0.0 0.0 0.0 0 0 08:59:13
hdisk66 5.2 49.5 11.3 776 2192 08:59:13
hdisk67 1.5 16.7 2.5 12 988 08:59:13
hdisk58 1.2 16.5 2.4 0 988 08:59:13
hdisk57 4.8 49.4 11.0 740 2224 08:59:13
hdisk68 0.1 0.5 0.1 4 24 08:59:13
hdisk61 0.2 1.6 0.3 0 96 08:59:13
hdisk70 0.0 0.7 0.1 0 44 08:59:13
cd1 0.0 0.0 0.0 0 0 08:59:13
cd0 0.0 0.0 0.0 0 0 08:59:13

tty: tin tout avg-cpu: % user % sys % idle % iowait time
0.0 0.0 15.8 13.5 47.0 23.7 09:00:13

Disks: % tm_act Kbps tps Kb_read Kb_wrtn time
hdisk1 4.9 57.0 12.3 276 3146 09:00:13
hdisk0 5.1 61.5 13.4 541 3146 09:00:13
hdisk49 29.9 416.4 90.0 8252 16728 09:00:13
hdisk51 0.1 0.5 0.1 32 0 09:00:13
hdisk47 0.3 3.6 0.9 0 216 09:00:13
hdisk43 0.2 1.5 0.3 0 92 09:00:13
hdisk53 0.9 17.7 3.5 152 912 09:00:13
hdisk46 0.0 0.0 0.0 0 0 09:00:13
hdisk42 0.0 0.0 0.0 0 0 09:00:13
hdisk39 5.9 48.1 11.5 1948 936 09:00:13
hdisk38 0.3 2.7 0.7 0 160 09:00:13
hdisk50 0.0 1.1 0.3 64 0 09:00:13
hdisk45 0.0 0.0 0.0 0 0 09:00:13
hdisk44 1.2 18.4 3.5 160 944 09:00:13
hdisk40 30.5 413.4 90.2 8328 16476 09:00:13
hdisk41 0.1 1.5 0.3 0 88 09:00:13
hdisk52 0.1 1.8 0.3 4 104 09:00:13
hdisk36 0.0 0.0 0.0 0 0 09:00:13
hdisk48 5.9 48.3 11.6 1900 1000 09:00:13
hdisk37 0.0 0.0 0.0 0 0 09:00:13
hdisk65 0.3 2.7 0.7 0 160 09:00:13
hdisk54 0.0 0.0 0.0 0 0 09:00:13
hdisk56 0.3 3.9 1.0 0 236 09:00:13
hdisk60 0.2 1.3 0.3 4 76 09:00:13
hdisk71 1.0 16.5 3.5 172 820 09:00:13
hdisk64 0.0 0.0 0.0 0 0 09:00:13
hdisk55 0.0 0.1 0.0 0 4 09:00:13
hdisk59 0.0 0.0 0.0 0 0 09:00:13
hdisk62 0.8 15.7 3.3 204 736 09:00:13
hdisk69 0.1 1.5 0.3 0 92 09:00:13
hdisk63 0.0 0.0 0.0 0 0 09:00:13
hdisk66 5.9 49.1 11.4 2000 948 09:00:13
hdisk67 29.8 417.9 90.3 8456 16616 09:00:13
hdisk58 30.4 412.8 89.8 7912 16852 09:00:13
hdisk57 5.9 48.9 11.5 1864 1072 09:00:13
hdisk68 0.2 1.4 0.2 8 76 09:00:13
hdisk61 0.1 1.6 0.3 0 96 09:00:13
hdisk70 0.1 1.6 0.3 0 96 09:00:13
cd1 0.0 0.0 0.0 0 0 09:00:13
cd0 0.0 0.0 0.0 0 0 09:00:13

tty: tin tout avg-cpu: % user % sys % idle % iowait time
0.0 0.0 13.4 8.3 53.7 24.6 09:01:13

Disks: % tm_act Kbps tps Kb_read Kb_wrtn time
hdisk1 3.6 154.9 8.9 7371 1920 09:01:13
hdisk0 2.9 33.5 8.0 87 1920 09:01:13
hdisk49 14.3 185.9 42.3 2572 8580 09:01:13
hdisk51 0.0 0.0 0.0 0 0 09:01:13
hdisk47 0.4 5.3 1.3 0 320 09:01:13
hdisk43 0.1 1.5 0.3 0 92 09:01:13
hdisk53 0.3 8.7 1.3 8 512 09:01:13
hdisk46 0.0 0.0 0.0 0 0 09:01:13
hdisk42 0.0 0.0 0.0 0 0 09:01:13
hdisk39 2.9 43.1 10.0 608 1976 09:01:13
hdisk38 0.6 5.3 1.3 0 316 09:01:13
hdisk50 20.9 266.5 63.8 5560 10428 09:01:13
hdisk45 0.0 0.0 0.0 0 0 09:01:13
hdisk44 0.5 25.5 1.4 1064 464 09:01:13
hdisk40 14.5 186.6 43.0 2720 8472 09:01:13
hdisk41 21.1 266.2 63.8 5724 10244 09:01:13
hdisk52 0.1 1.7 0.3 0 104 09:01:13
hdisk36 0.1 0.0 0.0 0 0 09:01:13
hdisk48 3.0 43.1 9.6 476 2112 09:01:13
hdisk37 0.0 0.0 0.0 0 0 09:01:13
hdisk65 0.4 5.3 1.3 0 320 09:01:13
hdisk54 0.0 0.0 0.0 0 0 09:01:13
hdisk56 0.5 5.0 1.3 0 300 09:01:13
hdisk60 0.0 0.0 0.0 0 0 09:01:13
hdisk71 0.5 10.7 1.5 0 640 09:01:13
hdisk64 0.0 0.0 0.0 0 0 09:01:13
hdisk55 0.1 17.6 0.2 1052 4 09:01:13
hdisk59 19.8 268.6 64.3 5760 10352 09:01:13
hdisk62 0.4 10.7 1.5 16 624 09:01:13
hdisk69 0.0 0.0 0.0 0 0 09:01:13
hdisk63 0.0 0.0 0.0 0 0 09:01:13
hdisk66 2.8 43.0 9.9 484 2096 09:01:13
hdisk67 13.7 188.0 43.0 2712 8564 09:01:13
hdisk58 14.1 185.0 43.2 2740 8360 09:01:13
hdisk57 3.0 43.6 9.9 576 2040 09:01:13
hdisk68 20.6 267.4 64.0 5660 10384 09:01:13
hdisk61 0.3 1.6 0.3 8 88 09:01:13
hdisk70 0.2 1.9 0.4 4 112 09:01:13
cd1 0.0 0.0 0.0 0 0 09:01:13
cd0 0.0 0.0 0.0 0 0 09:01:13

tty: tin tout avg-cpu: % user % sys % idle % iowait time
0.0 0.0 9.9 10.0 59.8 20.4 09:02:13

Disks: % tm_act Kbps tps Kb_read Kb_wrtn time
hdisk1 1.9 24.3 4.8 8 1450 09:02:13
hdisk0 2.8 28.9 5.9 284 1450 09:02:13
hdisk49 0.6 11.0 1.5 20 640 09:02:13
hdisk51 0.0 0.0 0.0 0 0 09:02:13
hdisk47 0.1 1.3 0.3 0 80 09:02:13
hdisk43 0.1 1.6 0.3 0 96 09:02:13
hdisk53 0.1 1.3 0.3 0 80 09:02:13
hdisk46 0.0 0.0 0.0 0 0 09:02:13
hdisk42 0.0 0.0 0.0 0 0 09:02:13
hdisk39 2.5 40.1 9.0 548 1856 09:02:13
hdisk38 0.1 1.3 0.3 0 80 09:02:13
hdisk50 27.9 336.6 77.7 6320 13876 09:02:13
hdisk45 0.0 0.0 0.0 0 0 09:02:13
hdisk44 0.2 0.8 0.2 4 44 09:02:13
hdisk40 0.3 10.0 1.3 4 596 09:02:13
hdisk41 27.8 338.7 77.8 6492 13832 09:02:13
hdisk52 0.1 1.2 0.2 0 72 09:02:13
hdisk36 0.1 0.0 0.0 0 0 09:02:13
hdisk48 2.9 41.7 9.3 576 1924 09:02:13
hdisk37 0.0 0.0 0.0 0 0 09:02:13
hdisk65 0.1 1.3 0.3 0 80 09:02:13
hdisk54 0.0 0.0 0.0 0 0 09:02:13
hdisk56 0.1 0.9 0.2 0 56 09:02:13
hdisk60 0.0 0.0 0.0 0 0 09:02:13
hdisk71 0.1 2.1 0.3 0 124 09:02:13
hdisk64 0.0 0.0 0.0 0 0 09:02:13
hdisk55 0.0 0.0 0.0 0 0 09:02:13
hdisk59 27.5 338.8 77.5 6284 14044 09:02:13
hdisk62 0.1 2.2 0.3 0 132 09:02:13
hdisk69 0.0 0.0 0.0 0 0 09:02:13
hdisk63 0.0 0.0 0.0 0 0 09:02:13
hdisk66 2.7 39.9 8.9 540 1856 09:02:13
hdisk67 0.7 8.6 1.3 8 508 09:02:13
hdisk58 0.6 10.0 1.4 4 596 09:02:13
hdisk57 2.9 39.3 8.9 532 1824 09:02:13
hdisk68 27.5 336.8 77.7 6360 13848 09:02:13
hdisk61 0.0 0.0 0.0 0 0 09:02:13
hdisk70 0.1 1.3 0.2 0 76 09:02:13
cd1 0.0 0.0 0.0 0 0 09:02:13
cd0 0.0 0.0 0.0 0 0 09:02:13

tty: tin tout avg-cpu: % user % sys % idle % iowait time
0.0 0.0 7.6 5.1 81.5 5.8 09:03:13

Disks: % tm_act Kbps tps Kb_read Kb_wrtn time
hdisk1 2.1 163.3 6.6 6319 3480 09:03:13
hdisk0 1.8 58.6 5.6 39 3480 09:03:13
hdisk49 0.9 14.5 2.4 124 748 09:03:13
hdisk51 0.0 0.0 0.0 0 0 09:03:13
hdisk47 0.1 1.0 0.2 0 60 09:03:13
hdisk43 0.1 0.9 0.2 0 56 09:03:13
hdisk53 0.2 3.8 0.6 16 212 09:03:13
hdisk46 0.0 0.0 0.0 0 0 09:03:13
hdisk42 0.0 0.0 0.0 0 0 09:03:13
hdisk39 4.1 26.6 6.3 1580 16 09:03:13
hdisk38 0.2 1.3 0.3 12 68 09:03:13
hdisk50 0.6 11.3 1.7 68 612 09:03:13
hdisk45 0.0 0.0 0.0 0 0 09:03:13
hdisk44 0.2 3.5 0.7 0 212 09:03:13
hdisk40 0.9 15.1 2.8 116 788 09:03:13
hdisk41 0.8 13.1 1.9 376 408 09:03:13
hdisk52 0.1 0.7 0.1 0 40 09:03:13
hdisk36 0.1 0.0 0.0 0 0 09:03:13
hdisk48 4.1 26.1 6.3 1516 52 09:03:13
hdisk37 0.0 0.0 0.0 0 0 09:03:13
hdisk65 0.1 1.3 0.3 0 80 09:03:13
hdisk54 0.0 0.0 0.0 0 0 09:03:13
hdisk56 0.0 0.8 0.2 0 48 09:03:13
hdisk60 0.0 0.0 0.0 0 0 09:03:13
hdisk71 0.1 1.8 0.4 0 108 09:03:13
hdisk64 0.2 1.2 0.3 72 0 09:03:13
hdisk55 0.1 0.3 0.1 16 4 09:03:13
hdisk59 0.4 9.9 1.4 160 436 09:03:13
hdisk62 0.1 1.3 0.3 0 80 09:03:13
hdisk69 0.0 0.0 0.0 0 0 09:03:13
hdisk63 0.0 0.0 0.0 0 0 09:03:13
hdisk66 4.2 28.1 6.6 1620 68 09:03:13
hdisk67 0.9 16.1 2.6 96 868 09:03:13
hdisk58 0.8 15.2 2.5 192 720 09:03:13
hdisk57 4.0 28.5 6.6 1592 116 09:03:13
hdisk68 0.6 11.7 1.9 284 416 09:03:13
hdisk61 0.2 1.8 0.3 0 108 09:03:13
hdisk70 0.1 1.8 0.3 0 108 09:03:13
cd1 0.0 0.0 0.0 0 0 09:03:13
cd0 0.0 0.0 0.0 0 0 09:03:13

tty: tin tout avg-cpu: % user % sys % idle % iowait time
0.0 0.0 11.6 6.9 79.6 1.9 09:04:13

Disks: % tm_act Kbps tps Kb_read Kb_wrtn time
hdisk1 1.9 18.6 4.2 4 1109 09:04:13
hdisk0 1.7 18.6 4.2 4 1109 09:04:13
hdisk49 0.5 6.1 1.1 4 364 09:04:13
hdisk51 0.0 0.0 0.0 0 0 09:04:13
hdisk47 0.0 0.3 0.1 0 20 09:04:13
hdisk43 0.1 2.7 0.5 0 160 09:04:13
hdisk53 0.1 1.6 0.4 0 96 09:04:13
hdisk46 0.0 0.0 0.0 0 0 09:04:13
hdisk42 0.0 0.0 0.0 0 0 09:04:13
hdisk39 1.7 25.2 5.3 264 1248 09:04:13
hdisk38 0.1 0.9 0.2 0 52 09:04:13
hdisk50 0.0 0.1 0.0 0 4 09:04:13
hdisk45 0.0 0.0 0.0 0 0 09:04:13
hdisk44 0.1 2.0 0.3 0 120 09:04:13
hdisk40 0.3 6.7 1.0 0 400 09:04:13
hdisk41 0.0 0.0 0.0 0 0 09:04:13
hdisk52 0.2 3.1 0.7 0 188 09:04:13
hdisk36 0.0 0.0 0.0 0 0 09:04:13
hdisk48 2.1 24.0 5.5 276 1164 09:04:13
hdisk37 0.0 0.0 0.0 0 0 09:04:13
hdisk65 0.0 1.3 0.3 0 80 09:04:13
hdisk54 0.0 0.0 0.0 0 0 09:04:13
hdisk56 0.1 1.3 0.3 0 80 09:04:13
hdisk60 0.0 0.0 0.0 0 0 09:04:13
hdisk71 0.1 2.9 0.4 0 172 09:04:13
hdisk64 0.0 0.0 0.0 0 0 09:04:13
hdisk55 0.0 0.0 0.0 0 0 09:04:13
hdisk59 0.1 1.5 0.3 0 92 09:04:13
hdisk62 0.1 4.0 0.6 0 240 09:04:13
hdisk69 0.0 0.0 0.0 0 0 09:04:13
hdisk63 0.0 0.0 0.0 0 0 09:04:13
hdisk66 2.0 23.3 5.2 240 1156 09:04:13
hdisk67 0.4 6.4 1.1 0 384 09:04:13
hdisk58 0.5 7.3 1.3 4 436 09:04:13
hdisk57 1.8 22.9 5.2 280 1092 09:04:13
hdisk68 0.0 0.5 0.1 0 32 09:04:13
hdisk61 0.1 1.6 0.3 0 96 09:04:13
hdisk70 0.1 1.5 0.3 0 92 09:04:13
cd1 0.0 0.0 0.0 0 0 09:04:13
cd0 0.0 0.0 0.0 0 0 09:04:13

tty: tin tout avg-cpu: % user % sys % idle % iowait time
0.0 0.0 17.8 11.9 66.7 3.6 09:05:13

Disks: % tm_act Kbps tps Kb_read Kb_wrtn time
hdisk1 1.9 122.3 5.2 6315 1021 09:05:13
hdisk0 1.6 17.1 4.2 7 1021 09:05:13
hdisk49 0.7 12.1 1.8 4 720 09:05:13
hdisk51 0.0 0.0 0.0 0 0 09:05:13
hdisk47 0.5 4.0 1.0 36 204 09:05:13
hdisk43 0.2 1.6 0.3 0 96 09:05:13
hdisk53 0.2 1.3 0.3 0 80 09:05:13
hdisk46 0.6 4.1 1.0 244 0 09:05:13
hdisk42 0.0 0.0 0.0 0 0 09:05:13
hdisk39 2.6 31.6 6.8 524 1372 09:05:13
hdisk38 0.3 3.1 0.8 4 184 09:05:13
hdisk50 0.1 1.3 0.3 80 0 09:05:13
hdisk45 0.0 0.0 0.0 0 0 09:05:13
hdisk44 0.2 2.3 0.5 0 136 09:05:13
hdisk40 0.6 13.7 1.7 0 820 09:05:13
hdisk41 0.1 1.9 0.3 0 116 09:05:13
hdisk52 0.2 1.7 0.3 0 104 09:05:13
hdisk36 0.0 0.0 0.0 0 0 09:05:13
hdisk48 2.6 28.1 6.4 448 1236 09:05:13
hdisk37 0.4 4.5 1.0 272 0 09:05:13
hdisk65 0.3 2.7 0.7 0 160 09:05:13
hdisk54 0.0 0.0 0.0 0 0 09:05:13
hdisk56 0.5 2.9 0.7 0 172 09:05:13
hdisk60 0.0 0.0 0.0 0 0 09:05:13
hdisk71 0.2 3.6 0.6 0 216 09:05:13
hdisk64 0.6 5.2 1.0 308 4 09:05:13
hdisk55 0.4 4.4 1.0 264 0 09:05:13
hdisk59 0.0 1.3 0.3 80 0 09:05:13
hdisk62 0.1 1.3 0.3 0 80 09:05:13
hdisk69 0.0 0.0 0.0 0 0 09:05:13
hdisk63 0.0 0.0 0.0 0 0 09:05:13
hdisk66 2.8 29.4 6.8 576 1188 09:05:13
hdisk67 0.8 11.1 1.7 0 664 09:05:13
hdisk58 0.6 10.1 1.7 4 604 09:05:13
hdisk57 3.1 29.5 6.7 516 1252 09:05:13
hdisk68 0.1 0.9 0.2 0 56 09:05:13
hdisk61 0.2 2.9 0.6 0 172 09:05:13
hdisk70 0.1 1.7 0.3 0 100 09:05:13
cd1 0.0 0.0 0.0 0 0 09:05:13
cd0 0.0 0.0 0.0 0 0 09:05:13