Unix File System performance with large directories

Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Unix File System performance with large directories
# 1  
Old 03-10-2004
Unix File System performance with large directories


how does the Unix File System perform with large directories (containing ~30.000 files)?
What kind of structure is used for the organization of a directory's content, linear lists, (binary) trees?

I hope the description 'Unix File System' is exact enough, I don't know more about the file system in use there...

Regards, Dirk
# 2  
Old 03-10-2004
Hi Dirk,

if you want to tune your filesystem, the most important question for you is , "What size are the files on it ?"

Related to this, you will change the blocksize and similar.

# 3  
Old 03-10-2004
Large directories devastate unix performance. With most file systems directories can grow but not shrink. So even a directory that once contained 30,000 files is a problem.

Directories are searched sequentially. They obviously must be searched to open a file. But they must also be scanned to find an empty slot during file creation. If the directory has subdirectories, a /bin/pwd in those subdirectories involves a search of the massive parent directory.

A large directory may be fragmented and scattered across the filesystem. This exacerbates the situation.

Unix commands such as ls are designed with smaller directories in mind. ls will read the entire directory into memory, sort it, then display the results. Even a ls executed in the directory will place a significant burden on the system.

uname -a
will tell you which version of unix you are using.

df -n /some/directory
will usually tell you the type of filesystem.
# 4  
Old 03-12-2004
I may be a bit fuzzy on this... but in HPUX, they no longer use blocksize.

Instead they use extent based sizing, default 4MB for Filesystem creation. Your files are based upon 4mb (default) intervals not 4k block multiples in HPUX... Especially where striping is involved.

Also, in Unices in general, everything is treated as a file so it is much easier to manage... I have a doc you can read which will shed a great deal of light for you...

read this section of this document...

The Art of Unix Programming
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script search, improve performance with large files

Hello, For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)
Discussion started by: SDohmen
15 Replies

2. HP-UX

Test cases for file system mount/umount performance in HP

Hi Folks, Could anyone please assist me with the what could be the scenarios to test the file system mount/umount performance check in HPUX. Thanks in advance, Vaishey (5 Replies)
Discussion started by: Vaishey
5 Replies

3. Shell Programming and Scripting

Performance issue in Grepping large files

I have around 300 files(*.rdf,*.fmb,*.pll,*.ctl,*.sh,*.sql,*.prog) which are of large size. Around 8000 keywords(which will be in the file $keywordfile) needed to be searched inside those files. If a keyword is found in a file..I have to insert the filename,extension,catagoery,keyword,occurrence... (8 Replies)
Discussion started by: millan
8 Replies

4. Red Hat

GFS file system performance is very slow

My code Hi All, I am having redhat linux 5.3 (Tikanga) with GFS file system and its very very slow for executing ls -ls command also.Please see the below for 2minits 12 second takes. Please help me to fix the issue. $ sudo time ls -la BadFiles |wc -l 0.01user 0.26system... (3 Replies)
Discussion started by: susindram
3 Replies

5. Red Hat

Empty directory, large size and performance

Hi, I've some directory that I used as working directory for a program. At the end of the procedure, the content is deleted. This directory, when I do a ls -l, appears to still take up some space. After a little research, I've seen on a another board of this forum that it's not really taking... (5 Replies)
Discussion started by: bdx
5 Replies

6. Programming

question about empty directories in unix system

how is it possible for a directory to be empty and still have a size greater than 0 in bytes... i made a shell script that shows info about all files/directories and this is what came up the last one is the size, here its showing 1024 in the for loop i did something like for h in * .*; do ... (4 Replies)
Discussion started by: omega666
4 Replies

7. Shell Programming and Scripting

Performance issue in UNIX while generating .dat file from large text file

Hello Gurus, We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this . Problem Definition: /Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below... (19 Replies)
Discussion started by: KRAMA
19 Replies

8. UNIX for Advanced & Expert Users

suggestions require for unix system performance on certain task

Dear all, On my UNIX server there is an apache web log file. The rate of logging of data in this file is very high. I want to extract user logging log from this file in run time. As soon as the user logging log logged in this file I want to redirect this user log into another file. I want to... (4 Replies)
Discussion started by: zing_foru
4 Replies

9. AIX

Why is my file system cache so large

Hi I have a filesystem cache which is around 20G in size and I'm a bit perplexed as to what is in it. I'm running Sybase on the machine with the db on raw volumes and a tempdb on a ramdisk. My understanding is that raw volumes are not cached and I assumed that the ramdisk is not either. Am... (1 Reply)
Discussion started by: mgibbons
1 Replies

10. UNIX for Dummies Questions & Answers

list directories on a file system

how can i see the list of directories, mounted on a filesystem? example, to show the list of directories mounted on / for aix (8 Replies)
Discussion started by: yls177
8 Replies
Login or Register to Ask a Question

Featured Tech Videos