We have an 8-processor Itanium system running HP-UX 11.23 connected to shared SAN discs.
We have an application that creates files (about 10) in a specific directory. When the application terminates, these files are removed (unlink) and a few others are updated. The directory contains 16000+ subdirectories and many other files. There can be up to 700 sessions running that application. Exiting the application usually takes 5-10 seconds. When a bunch of users (say 30) exit the application at the same time, it can take 1-2 minutes. We don't see slowdown during normal operations.
It seems like it's more noticeable as the number of files/subdirectories increases. In other words, it takes less time when there are less than 30000 files than there are more than 30000 files.
Glance and other performance tools do not report any bottlenecks. We couldn't find contention i.e. file locks in the application itself.
I have found a few places where they suggest keeping the number of directories in a path to a minimum. The directory above has only 3 levels e.g. /var/mydir/temp.
An HP-UX whitepaper also states:
Conversely, file access can be slowed when you have too many files (multiple thousands) in a given directory.
They don't provide any other details and could not find anything conclusive on the internet either.
We are suspecting contention at the file system or OS level when there are many concurrent create/remove requests.
Has anyone heard this before?
If so:
Can you explain what kind of performance problems this may cause?
Since there's activity at the directory level (unlink/rm), could there be contention in some file system table?
Is there a way to determine this is actually the case?
Aside from cleaning up the directory regularly, moving some files/directories to other locations, are there other alternatives?
Huge directories with many users can be a big bottleneck. A basic UFS directory is a huge unsorted and unstructured pile of name-inode# pairs, so access is by brute force. Create a tree and move files down.
PS: You can only shrink a directory, once you thin it out, by recreating it: make a sibling directory new/ right beside it (same file system so links are legal), clone the contents by cpio pass with use links (less space, faster), make directories and preserve times and permissions, then rename to swap it out, and later remove orig/:
I tried this on my older PARISC HP-UX, and the rm was problematic because it was on an nfs mount and a (script) file was open (even though linked elsewhere, so this is a bug in rm or nfs), so it left a .nfs#### file until it was closed. I used:
to identify the processes holding the file open.
Last edited by DGPickett; 11-05-2010 at 02:43 PM..
You say nothing about your SAN...
I noticed also this sort of issue (on a RP8400 16 CPU at the time...).
You dont give much information on accessed file size...and without any configuration described it will be difficult to help you but there is this Voltaire's quote:
"le mieux est l'ennemi du bien" I refer to when in such situation.
Which filesystem (FS) are the directories in? /var?
Which volume group (VG) is the FS in? vg00?
Where physically is the problem FS? internal disk or on the SAN?
Which disk is faster - internal or the SAN?
What else is going on in the problem FS and VG?
How big are these files? min, max, avg size.
If the problem is in /var on vg00 on internal disk, I would try making a new purpose built filesystem just for holding many thousands (of small, I'm guessing) files. You could try making a new VG and FS on the SAN (if it is faster), to hold this directory. At the very least this would isolate the issue.
Well check for big directories in this logical tree:
----------------
Recreating the directory tree has the advantage that directories will be low in every directory, shortening directory transit time. If a directory has many files and then a directory is added, it is a long search every time that directory is located. Of course, this points to the wisdom of keeping big collections of files in leaf directories, where nobody will be traversing.
Last edited by DGPickett; 11-11-2010 at 12:54 PM..
I want to thank everyone who took the time to respond. We haven't found a definitive answer to our problem. However, based on all the ideas and suggestions, we have been able to improve performance by splitting files into separate directories, reviewing the application to ensure files were removed when not needed anymore, etc.
Hi Experts,
Need some help with a script which is definetly beyond my scripting skills.
Here is flat file that I have with 4 Key Columns
KEYCOLUMN1 KEYCOLUMN2 KEYCOLUMN3 KEYCOLUMN4
123ABC AEG MANCHESTER BIGBOX... (2 Replies)
Hey,
I actually do have a question which seems rather easy for those you know more about this topic, since I am pretty new to bashscripting and don't know where it's limits are I have to ask you guys :)
Imagine a system where all possible code execution methods (binary executables or... (15 Replies)
I need to change how a posting procedure currently works in order to improve load balancing but I am hitting a potential file contention problem that I was wondering if someone here could assist me with...
In a directory called FilePool I would have a bunch of files that are constantly coming in... (3 Replies)
I need to write a script which will check all the directories in / and find out which is separate filesystem or a directory inside /
suppose i have
/application -->/dev/hdisk0 is a partitiom
/abc --> is a directory
/xyz-->is a directory
/db-->/dev/hdisk1 is a partition
actually my / gets... (2 Replies)
Hi All,
I want to find the percentage occupied by the directory in the file system.
Say, i have the file system
/home/arun/work
under this file system i have the directories
/home/arun/work/yesterday
/home/arun/work/today
/home/arun/work/tomorrow
The size of the file system is... (5 Replies)
Hi all,
Complete and utter virgin Unix person here (I don't even have the OS yet)
As I'm doing a "looking into it" kinda thing before I move from MS I hope my questions are not inappropriate.
1. Should I get some kind off anti virus software. I know Unix is pretty good for not getting them... (2 Replies)
I have a question and seek help. How many directory can be mounted on one file system on UNIX with solaris 9? For example, I have one file system as /dev/dsk/cieit0a6. I have created one directory as /u01/app/oracle and mounted this directory to cieit06. It works. Then I create another directory as... (4 Replies)
Can you give a very basic definition of file system vs. directory in unix. The context is around monitoring for disk capacity maxing out.
Why would /appl/sbe/inst_2/config/logs not be able to be because it is a directory
but appl/cce can be monitored because it is a unix file system. They... (1 Reply)