Sponsored Content
Special Forums Hardware Filesystems, Disks and Memory Performance Hit With Many Files Post 302293379 by otheus on Tuesday 3rd of March 2009 03:58:33 AM
Old 03-03-2009
Excellent question! It is indeed filesystem-dependent. Such performance metrics are hard to come by, since there are so many variables, and to do a good apples-apples comparison, you need the same computer with the same disks and install multiple OS's. But it may not be necessary, really, to get such statistics. One can look at the filesystem architecture, directory handling semantics, and conclude one might be better than the other.

Unforatunally, I cannot provide specifics on JFS nor NTFS. However, ReiserFS and modern versions of ext2 (on filesystems created with -O DIR_INDEX), file creation and lookup are very fast; they both use a hash index to find files. So if you know the name of the file, it can be found almost instantly (as I understand it and have experienced). On older versions of ext2, things really started to slow down after the directory entry itself extended to one or more indirect blocks -- maybe 1000 or so.

You do know one way to store large amounts of files is to create a directory hierarchy that is keyed on the filenames themselves? So files named "ergo1802.txt" might be stored in:
Code:
    data/er/go/18/ergo1802.txt

 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Anyone else see a performance hit from ext3

I reinstalled my Linux box with RedHat 7.2 and used the ext3 journaling file system. This thing is a pig now. There isn't much running on the box, and performance is sad. (1 Reply)
Discussion started by: 98_1LE
1 Replies

2. Shell Programming and Scripting

Shell Script to hit a url

Hi all, I have a php file that grabs xml, parses it and updates my db accordingly. I want to automate the execution of this process, rather than having to hit the url manually. I have been looking into using cron to execute a script to do this, however i'm not exactly sure what command i would... (1 Reply)
Discussion started by: restivz77
1 Replies

3. Programming

why multiple SIGINT raises when i hit C-c

hi, in my application, i have set up to capture SIGINT and execute a handler.the problem is whenever i hit C-c, multiple SIGINT are sent to the application.I have blocked the SIGINT right after catching the first one but it is unsuccessful.Here is what i do : jmp_buf main_loop; int... (1 Reply)
Discussion started by: Sedighzadeh
1 Replies

4. Shell Programming and Scripting

Hit count on a shell script

I have a unix shell script (ex.sh) written. How to find out how many users (incl. myself) have run this .sh ? I can insert code snipet at top of script if need be. - Ravi (2 Replies)
Discussion started by: ravi368
2 Replies

5. Shell Programming and Scripting

Getting Next Best Hit..

Hi.. I need to get the following output from the input file like this INPUT GRM1 GRM1 0 GRM1 ABC1 1 GRM1 FEQ1 2 GRM1 SED1 3 ABC2 GRM1 0 ABC2 ABC2 1 ABC2 FEQ1 2 ABC2 BED1 3 SED1 SED1 0 SED1 SED1 1 SED1 SED1 2 SED1 ABC1 3 OUTPUT: (7 Replies)
Discussion started by: empyrean
7 Replies

6. SuSE

Java hit

Hello, I'm having trouble looking for info for SUSIE on this CVE-2012-4681. This is basically the newest Java hit. It is mostly a web browser issue but I would like to see if the versions on our servers are vulnerable. I already found the pages/info for Solaris and RHEL. Any help would be... (4 Replies)
Discussion started by: bitlord
4 Replies

7. Cybersecurity

vnc password hit from Retina

Hello, I'm having an issue with VNC. Security at work says that they scanned my servers (Solaris, RHEL, SLES) and found that you don't need a password to access a VNC session. I have tested this and you can't login to the VNC session without a password. Can someone tell what the Retina scanner... (1 Reply)
Discussion started by: bitlord
1 Replies

8. Shell Programming and Scripting

Curl to hit the submit button

Hello, I am looking to hit a URL using curl and click on submit button so that I can get the results. The below is the code <input name="tos_accepted" id="tos_accepted" class="button" type="submit" value="Yes, I Agree"/> <input name="tos_discarded" id="tos_discarded"... (1 Reply)
Discussion started by: Kochappa
1 Replies

9. Shell Programming and Scripting

Best performance to merge two files

Hi Gurus, I need to merge two files. file1 (small file, only one line) this is first linefile2 (large file) abc def ghi ... I use below command to merge the file, since the file2 is really large file, the command read whole file2, the performance is not good. cat file1 > file3... (7 Replies)
Discussion started by: green_k
7 Replies
E2IMAGE(8)						      System Manager's Manual							E2IMAGE(8)

NAME
e2image - Save critical ext2/ext3/ext4 filesystem metadata to a file SYNOPSIS
e2image [ -rsI ] device image-file DESCRIPTION
The e2image program will save critical ext2, ext3, or ext4 filesystem metadata located on device to a file specified by image-file. The image file may be examined by dumpe2fs and debugfs, by using the -i option to those programs. This can assist an expert in recovering cat- astrophically corrupted filesystems. In the future, e2fsck will be enhanced to be able to use the image file to help recover a badly dam- aged filesystem. If image-file is -, then the output of e2image will be sent to standard output, so that the output can be piped to another program, such as gzip(1). (Note that this is currently only supported when creating a raw image file using the -r option, since the process of creating a normal image file, or QCOW2 image currently requires random access to the file, which cannot be done using a pipe. This restriction will hopefully be lifted in a future version of e2image.) It is a very good idea to create image files for all of filesystems on a system and save the partition layout (which can be generated using the fdisk -l command) at regular intervals --- at boot time, and/or every week or so. The image file should be stored on some filesystem other than the filesystem whose data it contains, to ensure that this data is accessible in the case where the filesystem has been badly damaged. To save disk space, e2image creates the image file as a sparse file, or in QCOW2 format. Hence, if the sparse image file needs to be copied to another location, it should either be compressed first or copied using the --sparse=always option to the GNU version of cp. This does not apply to the QCOW2 image, which is not sparse. The size of an ext2 image file depends primarily on the size of the filesystems and how many inodes are in use. For a typical 10 gigabyte filesystem, with 200,000 inodes in use out of 1.2 million inodes, the image file will be approximately 35 megabytes; a 4 gigabyte filesys- tem with 15,000 inodes in use out of 550,000 inodes will result in a 3 megabyte image file. Image files tend to be quite compressible; an image file taking up 32 megabytes of space on disk will generally compress down to 3 or 4 megabytes. RESTORING FILESYSTEM METADATA USING AN IMAGE FILE
The -I option will cause e2image to install the metadata stored in the image file back to the device. It can be used to restore the filesystem metadata back to the device in emergency situations. WARNING!!!! The -I option should only be used as a desperation measure when other alternatives have failed. If the filesystem has changed since the image file was created, data will be lost. In general, you should make a full image backup of the filesystem first, in case you wish to try other recovery strategies afterwards. RAW IMAGE FILES
The -r option will create a raw image file instead of a normal image file. A raw image file differs from a normal image file in two ways. First, the filesystem metadata is placed in the proper position so that e2fsck, dumpe2fs, debugfs, etc. can be run directly on the raw image file. In order to minimize the amount of disk space consumed by a raw image file, the file is created as a sparse file. (Beware of copying or compressing/decompressing this file with utilities that don't understand how to create sparse files; the file will become as large as the filesystem itself!) Secondly, the raw image file also includes indirect blocks and directory blocks, which the standard image file does not have, although this may change in the future. Raw image files are sometimes used when sending filesystems to the maintainer as part of bug reports to e2fsprogs. When used in this capacity, the recommended command is as follows (replace hda1 with the appropriate device): e2image -r /dev/hda1 - | bzip2 > hda1.e2i.bz2 This will only send the metadata information, without any data blocks. However, the filenames in the directory blocks can still reveal information about the contents of the filesystem that the bug reporter may wish to keep confidential. To address this concern, the -s option can be specified. This will cause e2image to scramble directory entries and zero out any unused portions of the directory blocks before writing the image file. However, the -s option will prevent analysis of problems related to hash-tree indexed directories. Note that this will work even if you substitute "/dev/hda1" for another raw disk image, or QCOW2 image previously created by e2image. QCOW2 IMAGE FILES The -Q option will create a QCOW2 image file instead of a normal, or raw image file. A QCOW2 image contains all the information the raw image does, however unlike the raw image it is not sparse. The QCOW2 image minimize the amount of disk space by storing data in special format with pack data closely together, hence avoiding holes while still minimizing size. In order to send filesystem to the maintainer as a part of bug report to e2fsprogs, use following commands (replace hda1 with the appropri- ate device): e2image -Q /dev/hda1 hda1.qcow2 bzip2 -z hda1.qcow2 This will only send the metadata information, without any data blocks. However, the filenames in the directory blocks can still reveal information about the contents of the filesystem that the bug reporter may wish to keep confidential. To address this concern, the -s option can be specified. This will cause e2image to scramble directory entries and zero out any unused portions of the directory blocks before writing the image file. However, the -s option will prevent analysis of problems related to hash-tree indexed directories. Note that QCOW2 image created by e2image is regular QCOW2 image and can be processed by tools aware of QCOW2 format such as for example qemu-img. AUTHOR
e2image was written by Theodore Ts'o (tytso@mit.edu). AVAILABILITY
e2image is part of the e2fsprogs package and is available from http://e2fsprogs.sourceforge.net. SEE ALSO
dumpe2fs(8), debugfs(8) E2fsprogs version 1.42.5 July 2012 E2IMAGE(8)
All times are GMT -4. The time now is 05:02 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy