Performance Hit With Many Files Post: 302293379

Sponsored Content

Special Forums Hardware Filesystems, Disks and Memory Performance Hit With Many Files Post 302293379 by otheus on Tuesday 3rd of March 2009 03:58:33 AM

03-03-2009

Registered User

Excellent question! It is indeed filesystem-dependent. Such performance metrics are hard to come by, since there are so many variables, and to do a good apples-apples comparison, you need the same computer with the same disks and install multiple OS's. But it may not be necessary, really, to get such statistics. One can look at the filesystem architecture, directory handling semantics, and conclude one might be better than the other.

Unforatunally, I cannot provide specifics on JFS nor NTFS. However, ReiserFS and modern versions of ext2 (on filesystems created with -O DIR_INDEX), file creation and lookup are very fast; they both use a hash index to find files. So if you know the name of the file, it can be found almost instantly (as I understand it and have experienced). On older versions of ext2, things really started to slow down after the directory entry itself extended to one or more indirect blocks -- maybe 1000 or so.

You do know one way to store large amounts of files is to create a directory hierarchy that is keyed on the filenames themselves? So files named "ergo1802.txt" might be stored in:

Code:

    data/er/go/18/ergo1802.txt

otheus

View Public Profile for otheus

Find all posts by otheus

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Anyone else see a performance hit from ext3

I reinstalled my Linux box with RedHat 7.2 and used the ext3 journaling file system. This thing is a pig now. There isn't much running on the box, and performance is sad.

2. Shell Programming and Scripting

Shell Script to hit a url

Hi all, I have a php file that grabs xml, parses it and updates my db accordingly. I want to automate the execution of this process, rather than having to hit the url manually. I have been looking into using cron to execute a script to do this, however i'm not exactly sure what command i would...

3. Programming

why multiple SIGINT raises when i hit C-c

hi, in my application, i have set up to capture SIGINT and execute a handler.the problem is whenever i hit C-c, multiple SIGINT are sent to the application.I have blocked the SIGINT right after catching the first one but it is unsuccessful.Here is what i do : jmp_buf main_loop; int...

4. Shell Programming and Scripting

Hit count on a shell script

I have a unix shell script (ex.sh) written. How to find out how many users (incl. myself) have run this .sh ? I can insert code snipet at top of script if need be. - Ravi

5. Shell Programming and Scripting

Getting Next Best Hit..

Hi.. I need to get the following output from the input file like this INPUT GRM1 GRM1 0 GRM1 ABC1 1 GRM1 FEQ1 2 GRM1 SED1 3 ABC2 GRM1 0 ABC2 ABC2 1 ABC2 FEQ1 2 ABC2 BED1 3 SED1 SED1 0 SED1 SED1 1 SED1 SED1 2 SED1 ABC1 3 OUTPUT:

6. SuSE

Java hit

Hello, I'm having trouble looking for info for SUSIE on this CVE-2012-4681. This is basically the newest Java hit. It is mostly a web browser issue but I would like to see if the versions on our servers are vulnerable. I already found the pages/info for Solaris and RHEL. Any help would be...

7. Cybersecurity

vnc password hit from Retina

Hello, I'm having an issue with VNC. Security at work says that they scanned my servers (Solaris, RHEL, SLES) and found that you don't need a password to access a VNC session. I have tested this and you can't login to the VNC session without a password. Can someone tell what the Retina scanner...

8. Shell Programming and Scripting

Curl to hit the submit button

Hello, I am looking to hit a URL using curl and click on submit button so that I can get the results. The below is the code <input name="tos_accepted" id="tos_accepted" class="button" type="submit" value="Yes, I Agree"/> <input name="tos_discarded" id="tos_discarded"...

9. Shell Programming and Scripting

Best performance to merge two files

Hi Gurus, I need to merge two files. file1 (small file, only one line) this is first linefile2 (large file) abc def ghi ... I use below command to merge the file, since the file2 is really large file, the command read whole file2, the performance is not good. cat file1 > file3...

LEARN ABOUT DEBIAN

jfs_logdump

jfs_logdump(8)						  JFS utility - journal log dump					    jfs_logdump(8)

NAME

       jfs_logdump - dump a JFS formatted device's journal log

SYNOPSIS

       jfs_logdump [ -a ] device

DESCRIPTION

       jfs_logdump dumps the contents of the journal log from the specified JFS formatted device into output file ./jfslog.dmp.

       device is the special file name corresponding to the actual device from which the journal log will be dumped (e.g.  /dev/hdb1 ).

       jfs_logdump must be run as root.

OPTIONS

       -a     Dump the entire contents of the journal log instead of just the committed transactions since the last synch point.

EXAMPLE

       Dump the entire JFS journal log from the 3rd partition of the 2nd hard disk into ./jfslog.dmp:

	      jfs_logdump -a /dev/hdb3

REPORTING BUGS

       If you find a bug in JFS or jfs_logdump, please report it via the bug tracking system ("Report Bugs" section) of the JFS project web site:
       http://jfs.sourceforge.net/

       Please send as much pertinent information as possible, including any jfs_logdump error messages and the complete output of running jfs_fsck
       with the -v option on the JFS device.

SEE ALSO

       jfs_fsck(8), jfs_mkfs(8), jfs_fscklog(8), jfs_tune(8), jfs_debugfs(8)

AUTHORS

       Barry Arndt  (barndt@us.ibm.com)
       William Braswell, Jr.

       jfs_logdump is maintained by IBM.
       See the JFS project web site for more details:
       http://jfs.sourceforge.net/

								 October 29, 2002						    jfs_logdump(8)