Visit The New, Modern Unix Linux Community


Advice regarding filesystems handling large number of files

 
Thread Tools Search this Thread
Operating Systems Linux Red Hat Advice regarding filesystems handling large number of files
# 1  
Advice regarding filesystems handling large number of files

Hi All,

I have a CentOS operating system installed. I work with really huge number of files which are not only huge in number but some of them really huge in size. Minimum number of files could be 1 million to 2 million in one directory itself. Some of the files are even several Gigabytes in size. Like 10Gb 15Gb etc.

I have ext3 filesystem on my disk drive. Recently, that disk drive crashed. I am not sure why it crashed. But when I used to work and execute programs the hard disk used to take lots of time to respond. Even find and du -h commands used to take lots of time to show results.

I would request any one of you to kindly advice me as to why the disk drive crashed?

Is it because:

1. The filesystem ext3 is not suited for such large number of files?

2. Is it because there are too many files and I should have organized those files in several directories to reduce the load on Hard Disk's cylinders?

3. I've read that XFS file system is well catered to large number of files. Should I use XFS filesystem on my new hard drive?

4. The hard disk should be around 3 years old.

The machine is really good with 50GB RAM and 16 core processor.
# 2  
EXT3 is good only if not have anything from production. For serious purpose I'm going to recommend XFS because I have good experience with that FS.
This User Gave Thanks to solaris_user For This Post:
# 3  
I'd also recommend XFS. Just be very careful - make SURE your system is up-to-date, and if you're building your filesystem(s) on top of LVM, make doggone sure your kernel's LVM implementation supports file system barriers.

XFS kind of EXPECTS its barrier requests to work, not be silently ignored like LVM used to do.

Just Google "Linux file system barriers".

(Actually, it's the Linux devmapper underneath LVM that used to silently ignore barrier requests - given how long Linux has been around, that's only been fixed very recently...)
This User Gave Thanks to achenle For This Post:

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #108
Difficulty: Easy
The outer layers of the Linux architecture has the shell, shell commands and application programs.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Advice on how to set up error handling

Hi Folks - I want to add error handling to a portion of a *.ksh, but I'm having difficulty doing so in an easily digestible way. Essentially, I want to echo weather it was successful or unsuccessful after each command. Here is the code I need to add error handling to: perl... (2 Replies)
Discussion started by: SIMMS7400
2 Replies

2. Shell Programming and Scripting

Sftp large number of files

Want to sftp large number of files ... approx 150 files will come to server every minute. (AIX box) Also need make sure file has been sftped successfully... Please let me know : 1. What is the best / faster way to transfer files? 2. should I use batch option -b so that connectivity will be... (3 Replies)
Discussion started by: vegasluxor
3 Replies

3. UNIX for Dummies Questions & Answers

Rename a large number of files in subdirectories

Hi, I have a large number of subdirectories (>200), and in each of these directories there is a file with a name like "opp1234.dat". I'd like to know how I could change the names of these files to say "out.dat" in all these subdirectories in one go. Thanks! (5 Replies)
Discussion started by: lost.identity
5 Replies

4. SCO

Need advice: Copying large CSV report files off SCO system

I have a SCO Unix server from 1999 running SCO 5.0.5 and some ancient accounting software called Real World A report writer program on the system is used to generate CSV files from accounting that we write with DOSCOPY commands to 3.5" floppies In the next 60 days we will be decommissioning... (11 Replies)
Discussion started by: magnetman
11 Replies

5. UNIX for Dummies Questions & Answers

Delete large number of files

Hi. I need to delete a large number of files listed in a txt file. There are over 90000 files in the list. Some of the directory names and some of the file names do have spaces in them. In the file, each line is a full path to a file: /path/to/the files/file1 /path/to/some other/files/file 2... (4 Replies)
Discussion started by: inakajin
4 Replies

6. Shell Programming and Scripting

Concatenation of a large number of files

Hellow i have a large number of files that i want to concatenate to one. these files start with the word 'VOICE_' for example VOICE_0000000000 VOICE_1223o23u0 VOICE_934934927349 I use the following code: cat /ODS/prepaid/CDR_FLOW/MEDIATION/VOICE_* >> /ODS/prepaid/CDR_FLOW/WORK/VOICE ... (10 Replies)
Discussion started by: chriss_58
10 Replies

7. Solaris

How to safely copy full filesystems with large files (10Gb files)

Hello everyone. Need some help copying a filesystem. The situation is this: I have an oracle DB mounted on /u01 and need to copy it to /u02. /u01 is 500 Gb and /u02 is 300 Gb. The size used on /u01 is 187 Gb. This is running on solaris 9 and both filesystems are UFS. I have tried to do it using:... (14 Replies)
Discussion started by: dragonov7
14 Replies

8. UNIX for Dummies Questions & Answers

questing regarding tar large number of files

I want to tar large number of files about 150k. i am using the find command as below to create a file with all file names. & then trying to use the tar -I command as below. # find . -type f -name "gpi*" > include-file # tar -I include-file -cvf newfile.tar This i got from one of the posts... (2 Replies)
Discussion started by: crux123
2 Replies

9. Shell Programming and Scripting

Error Handling -pls advice

Dear friends, I am using the below command in my unix script ----------------------------------------------- File_Name=`ls $CTRY*$DATE_SUFFIX*zip` --> Command-1 ..... if then unzip -a $File_Name -d $CTRY_DIR/$CTRY else echo "File for $CTRY dated $DATE_SUFFIX does not exist... (2 Replies)
Discussion started by: sureshg_sampat
2 Replies

10. Shell Programming and Scripting

moving large number of files

I have a task to move more than 35000 files every two hours, from the same directory to another directory based on a file that has the list of filenames I tried the following logics (1) find . -name \*.dat > list for i in `cat list` do mv $i test/ done (2) cat list|xargs -i mv "{}"... (7 Replies)
Discussion started by: bryan
7 Replies

Featured Tech Videos