Filesystem mystery: disks are not busy on one machine, very busy on a similar box Post: 302343494

Sponsored Content

Top Forums UNIX for Advanced & Expert Users Filesystem mystery: disks are not busy on one machine, very busy on a similar box Post 302343494 by mschwage on Thursday 13th of August 2009 01:56:59 PM

08-13-2009

Registered User

Quote:

Originally Posted by jim mcnamara

Are the machines reading/writing the same directories?. Directory size can really affect performance of ls and other file operations.

Are the files mounted with NSF? If so are the mountpoints off the root directory / ?

IO request queue lengths are huge on the bad box as well.

Thank for the reply. Here's my replies:

No they are not the same size. For the bad machine, the directories were 10 times as large. I have asked the developer to tune his application so the directory sizes are comparable.
No they are not mounted on NFS. They are on locally-mounted hardware RAID5 partition.
Yes, I expect queue lengths to be long since the machine is backed up writing to the disk.

Now the developer has reduced the number of files being written to. I have run these commands in the directories where the files live:

Code:

touch /tmp/outfile; sleep 10 ; find . -newer /tmp/outfile | wc -l

This will show me how many files have been written to in the last 10 seconds. On the good machine, I am writing to 3000 files in that time period. On the bad machine, I am writing to 125 files. This is an average, more or less.

Now I am still seeing this on the bad machine:

Code:

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d1        7.00   180.00 30.00 323.00  3712.00  4032.00    21.94    66.76  189.12   2.18  76.80

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d1        0.00  2004.00  7.00 171.00   896.00  8744.00    54.16    73.36  201.26   2.92  52.00

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d1        0.00  2445.00  2.00 237.00   256.00 24112.00   101.96   121.27  539.90   4.19 100.10

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d1        8.00   147.00 85.00 50.00 10760.00  7968.00   138.73     3.45  246.15   2.38  32.10

Note that at 100.10% utilization, we're doing only writes, and the "await" value is very high- 500 ms. or so. It seems that writes are killing us for some reason.
-Mike

---------- Post updated 08-13-09 at 12:56 PM ---------- Previous update was 08-12-09 at 02:52 PM ----------

OK, we figured it out: our partition is a RAID-5 controlled by an HP P400i RAID controller. We turned on the Drive Write Cache, but the Array Accelerator needs to be turned on for *each logical drive*. We only had it on for drive 1. Our problematic partition is logical drive 2. Once we turned it on, using

Code:

hpacucli ctrl slot=0 ld 2 modify arrayaccelerator=enable

we saw much much much better iostat results. Using the same iostat command as above, we now have:

Code:

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d1        0.00   173.20  0.00 77.40     0.00  2008.00    25.94     0.02    0.26   0.02   0.12

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d1        0.00   512.00  0.00 95.00     0.00  4856.00    51.12     0.02    0.22   0.03   0.30

mschwage

View Public Profile for mschwage

Find all posts by mschwage

4 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

umount, device busy, but..

I'm trying to unmount a file system, but umount says the device is busy. A fuser -c /myfs tells me that nothing on this fs is in use. Any idea? Oh, and btw, why isn't my signature beeing displayed? Show user's signature is enabled and I have entered one :/

2. UNIX for Advanced & Expert Users

CD Drive is busy

one problem which seems simple but i've faced many times is when i've used cd & want to eject, however i'm not in /dev/cdrom/... in any of my Xterms but it gives the message that the the CD drive is busy. the only thing i could do is /etc/init.d/volmgt stop & eject the CD manually but it doesn't...

3. HP-UX

Cannot umount - device busy

Anyone have any idea why I cannot umount a directory even though fuser shows no process attached to it? fuser -cu /data/oracle/GMPSHRDM/export /data/oracle/GMPSHRDM/export: umount /data/oracle/GMPSHRDM/export umount: cannot unmount /data/oracle/GMPSHRDM/export : Device busy umount:...

4. Programming

using signals instead of busy wait

Hello, First time posting on this board hopefully someone will able to help me. I am looking for some examples of c programs which use a signal to notify the program to begin a certain action instead of having it continually loop to check to see if condition is attained(busy wait). Any...

4 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

umount, device busy, but..

Discussion started by: sTorm

2. UNIX for Advanced & Expert Users

CD Drive is busy

Discussion started by: nikk

3. HP-UX

Cannot umount - device busy

Discussion started by: keelba

4. Programming

using signals instead of busy wait

Discussion started by: jut26