HP-UX

We have an 8-processor Itanium system running HP-UX 11.23 connected to shared SAN discs.

We have an application that creates files (about 10) in a specific directory. When the application terminates, these files are removed (unlink) and a few others are updated. The directory contains 16000+ subdirectories and many other files. There can be up to 700 sessions running that application. Exiting the application usually takes 5-10 seconds. When a bunch of users (say 30) exit the application at the same time, it can take 1-2 minutes. We don't see slowdown during normal operations.



It seems like it's more noticeable as the number of files/subdirectories increases. In other words, it takes less time when there are less than 30000 files than there are more than 30000 files.

Glance and other performance tools do not report any bottlenecks. We couldn't find contention i.e. file locks in the application itself.

I have found a few places where they suggest keeping the number of directories in a path to a minimum. The directory above has only 3 levels e.g. /var/mydir/temp.

An HP-UX whitepaper also states:

Conversely, file access can be slowed when you have too many files (multiple thousands) in a given directory.

They don't provide any other details and could not find anything conclusive on the internet either.


We are suspecting contention at the file system or OS level when there are many concurrent create/remove requests.



Has anyone heard this before?
If so:
Can you explain what kind of performance problems this may cause?
Since there's activity at the directory level (unlink/rm), could there be contention in some file system table?
Is there a way to determine this is actually the case?
Aside from cleaning up the directory regularly, moving some files/directories to other locations, are there other alternatives?

Huge directories with many users can be a big bottleneck. A basic UFS directory is a huge unsorted and unstructured pile of name-inode# pairs, so access is by brute force. Create a tree and move files down.

PS: You can only shrink a directory, once you thin it out, by recreating it: make a sibling directory new/ right beside it (same file system so links are legal), clone the contents by cpio pass with use links (less space, faster), make directories and preserve times and permissions, then rename to swap it out, and later remove orig/:

Code:

mkdir new
cd old_dir
find . |cpio -pdal ../new
cd ..
mv old_dir orig
mv new old_dir
#later
rm -rf orig

I tried this on my older PARISC HP-UX, and the rm was problematic because it was on an nfs mount and a (script) file was open (even though linked elsewhere, so this is a bug in rm or nfs), so it left a .nfs#### file until it was closed. I used:

Code:

fuser <file> <dir>
ps -fp <pid1>,<pid2>

to identify the processes holding the file open.

You say nothing about your SAN...
I noticed also this sort of issue (on a RP8400 16 CPU at the time...).

You dont give much information on accessed file size...and without any configuration described it will be difficult to help you but there is this Voltaire's quote:
"le mieux est l'ennemi du bien" I refer to when in such situation.

Some thoughts that come to mind:

Which filesystem (FS) are the directories in? /var?
Which volume group (VG) is the FS in? vg00?
Where physically is the problem FS? internal disk or on the SAN?

Which disk is faster - internal or the SAN?
What else is going on in the problem FS and VG?
How big are these files? min, max, avg size.

If the problem is in /var on vg00 on internal disk, I would try making a new purpose built filesystem just for holding many thousands (of small, I'm guessing) files. You could try making a new VG and FS on the SAN (if it is faster), to hold this directory. At the very least this would isolate the issue.

Well check for big directories in this logical tree:

Code:

$ find * -type d -follow | xargs -n999 ls -ld | sort -nr +4 -5 | head

----------------
Recreating the directory tree has the advantage that directories will be low in every directory, shortening directory transit time. If a directory has many files and then a directory is added, it is a long search every time that directory is located. Of course, this points to the wisdom of keeping big collections of files in leaf directories, where nobody will be traversing.

So like UNIX: many options, some for light duty, some for heavy duty use.

I want to thank everyone who took the time to respond. We haven't found a definitive answer to our problem. However, based on all the ideas and suggestions, we have been able to improve performance by splitting files into separate directories, reviewing the application to ensure files were removed when not needed anymore, etc.