Processing very big directory


 
Thread Tools Search this Thread
Operating Systems HP-UX Processing very big directory
# 1  
Old 06-04-2009
Processing very big directory

Hello,

I have a very big directory with lots of small files. Just doing a single "ls" takes eternity, so I even can't know how many files the directory has.

I need to get the newest files in the last 15 minutes, but I can't get an eficient way of doing it. I've tried find, perl, shell script, etc. Is there any more eficient way of doing this? Maybe avoiding reading all the files, or going to the end of the directory...

PS: Using HP-UX B.11.31 U ia64

Any help?
Thanks.
# 2  
Old 06-04-2009
To count the number of files in the directory. A "find" does not sort the output and will be quicker than "ls".

Code:
find . -type f -print|wc -l

To find out the size of the directory file itself:

Code:
ls -lad .

A directory file above one megabyte is pretty large. I would expect say 25,000 files in a megabyte size directory.
The performance of a mature directory can be improved by periodically re-creating the directory file.
# 3  
Old 06-04-2009
Well, my directory file as 180MB ...

Code:
$ ls -lad .
drwxrwxr-x  10 interf     sapsys     189300736 Jun  4 17:57 .
$

# 4  
Old 06-04-2009
There is no really easy to do it other than finding a more efficient data storage structure.

However it you are using vxfs and are deleting files are an amount of time has elapsed you could try a directory defrag to see if the directory is bigger than it needs to be and reduce it if possible. The is not the current number of files but the highest number ever that matters.
# 5  
Old 06-05-2009
Any luck with finding out how many files you have?
What processing you you need to do in the directory?
Is archiving an option?

The inode count is the total number of files and directories on a filesystem. It is possible to run out of inodes long before you fill a filesystem and well worth being aware of how many inodes you have left:

Code:
bdf -i

Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete big directory issue

Hello folks, I am deleting a directory with script it is taking 11Hour and also increase the IO on server. I am using below command, inside date directory there are hour directories, which i am deleting after archiving. Archiving is not taking long time, only "rm -rf" is taking alot of time with... (21 Replies)
Discussion started by: learnbash
21 Replies

2. HP-UX

Empty Directory, big size

Hello, Can you please explain why I have various empty directories with large size ? OS is B.11.11 (3 Replies)
Discussion started by: drbiloukos
3 Replies

3. UNIX for Dummies Questions & Answers

Empty Directory, big size

Hello, Can somebody please explain why I have EMPTY directories on HP-UX with BIG SIZE ? Thank you ! Double post, continued here (0 Replies)
Discussion started by: drbiloukos
0 Replies

4. Shell Programming and Scripting

Find which dir is big size in a directory

Hi all, Could you please tellme the commadn which sorts the list of directories in a parent dir by their size. Thanks. (2 Replies)
Discussion started by: firestar
2 Replies

5. AIX

Fastest way to count big amount of files in sub directory

Hi, what happened is we want to count all the files in a directory and inside this directory got many folders and so take long time to count it. Already run for about few minutes but still not done. The command we use to count is find . -type f | wc -l Just wondering if there is any other... (9 Replies)
Discussion started by: ngaisteve1
9 Replies

6. UNIX for Advanced & Expert Users

mv OR cp many files from 1 big directory with sub folders all to 1 spot?

Hello All. I am trying to do this from a terminal prompt on my mac.... I have 100 folders all named different things. Those 100 folders are inside ~/Desktop/Pictures directory. Each of the 100 folders are uniquely named. The image files inside of each folder only have some similarities. ... (1 Reply)
Discussion started by: yoyoyo777
1 Replies

7. Shell Programming and Scripting

how to extract files one by one from a directory and let some processing happen

how to extract files one by one from a directory and let some processing be done on the file I have a directory by name INTRN which has files like INTR.0003080248636814 INTR.0003080248636816 INTR.0003080248636818 . . . . and so on and in a script... (5 Replies)
Discussion started by: saniya
5 Replies

8. Shell Programming and Scripting

Processing files within a directory one by one

Hi How to create a shell script which takes in to account all the files present within a directory DIR one by one e.g. suppose i have a directory named DIR where there are files with the extension .ABC i want to create shell script which processes all these files one by one. ... (1 Reply)
Discussion started by: skyineyes
1 Replies

9. Shell Programming and Scripting

Need script to make big directory structure

Hi, I'm a novice and I'd like to make a directory structure with a hundred or so folders. I've tried mkdir /foo/foo1/etc... mkdir /foo/foo2/etc mkdir /foo/foo3/etc mkdir /foo/foo4/etc ...but it appends '@' to each folder name and then fails on the subdirectories. Is it better to use a... (2 Replies)
Discussion started by: kamur
2 Replies
Login or Register to Ask a Question