UNIX for Dummies Questions & Answers

Hi,

how does the Unix File System perform with large directories (containing ~30.000 files)?
What kind of structure is used for the organization of a directory's content, linear lists, (binary) trees?

I hope the description 'Unix File System' is exact enough, I don't know more about the file system in use there...

Regards, Dirk

Hi Dirk,

if you want to tune your filesystem, the most important question for you is , "What size are the files on it ?"

Related to this, you will change the blocksize and similar.

Regards
Malcom

Large directories devastate unix performance. With most file systems directories can grow but not shrink. So even a directory that once contained 30,000 files is a problem.

Directories are searched sequentially. They obviously must be searched to open a file. But they must also be scanned to find an empty slot during file creation. If the directory has subdirectories, a /bin/pwd in those subdirectories involves a search of the massive parent directory.

A large directory may be fragmented and scattered across the filesystem. This exacerbates the situation.

Unix commands such as ls are designed with smaller directories in mind. ls will read the entire directory into memory, sort it, then display the results. Even a ls executed in the directory will place a significant burden on the system.

uname -a
will tell you which version of unix you are using.

df -n /some/directory
will usually tell you the type of filesystem.

I may be a bit fuzzy on this... but in HPUX, they no longer use blocksize.

Instead they use extent based sizing, default 4MB for Filesystem creation. Your files are based upon 4mb (default) intervals not 4k block multiples in HPUX... Especially where striping is involved.


Also, in Unices in general, everything is treated as a file so it is much easier to manage... I have a doc you can read which will shed a great deal of light for you...

read this section of this document...

The Art of Unix Programming