Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Fastest way to list a file in a folder containing 800,000 files using wildcard

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Fastest way to list a file in a folder containing 800,000 files using wildcard
# 1  
Old 07-09-2008
Fastest way to list a file in a folder containing 800,000 files using wildcard
Hi,

I have a directory with possibly around 800,000 files in it.
What is the fastest way to list file(s) in this directory with a wildcard.

for example would

Code:
ls -1 *.abcdefg.Z

or

Code:
find . -name "*.abcdefg.Z"


be the fastest way to find all of the files that end with .abcdefg.Z in this directory with 800,000 files in it.
Any help or sugestion would be greatly appreciated.

Thx
Jerardfjay
Last edited by jerardfjay; 07-09-2008 at 01:07 PM.. Reason: better description of issue
# 2  
Old 07-09-2008
Try them out with "time command":

Code:
time ls -1 *.abcdefg.Z

Code:
time find . -name "*.abcdefg.Z"

Regards
# 3  
Old 07-09-2008
ls -1 wins
Looks like ls -1 is faster than find command.
Here are the results.

Code:
time ls -1 *.00abcdefg.Z
ABC.00abcdefg.Z
AB.00abcdefg.Z

real    0m2.05s
user    0m0.88s
sys     0m0.96s

Code:
 time find . -name "*.00abcdefg.Z"
./ABC.00abcdefg.Z
./AB.00abcdefg.Z

real    33m43.95s
user    0m3.02s
sys     0m29.87s

I am a bit surprised, since I would have imagined the find to be faster. However if there is some other mechanism other than ls -1 available I would be open to trying that as well. Thanks for your input.

Jerardfjay
# 4  
Old 07-09-2008
Quote:
I am a bit surprised, since I would have imagined the find to be faster. However if there is some other mechanism other than ls -1 available I would be open to trying that as well.
Assuming no IFS characters in the filenames (otherwise you should change the IFS before running this command):

Code:
printf "%s\n" *.00abcdefg.Z

# 5  
Old 07-09-2008
Quote:
Originally Posted by jerardfjay
Looks like ls -1 is faster than find command.
Here are the results.

Code:
time ls -1 *.00abcdefg.Z
ABC.00abcdefg.Z
AB.00abcdefg.Z

real    0m2.05s
user    0m0.88s
sys     0m0.96s

Code:
 time find . -name "*.00abcdefg.Z"
./ABC.00abcdefg.Z
./AB.00abcdefg.Z

real    33m43.95s
user    0m3.02s
sys     0m29.87s

I am a bit surprised, since I would have imagined the find to be faster. However if there is some other mechanism other than ls -1 available I would be open to trying that as well. Thanks for your input.

Jerardfjay
can you check this and tell the time taken?
time find . -name "*.00abcdefg.Z" -maxdepth 1 -type f
# 6  
Old 07-09-2008
Quote:
Originally Posted by Franklin52
Try them out with "time command":

Code:
time ls -1 *.abcdefg.Z


You type that command into an interactive shell. Your interactive shell will see *.abcdefg.Z and do all of the work of searching the directory to expand that pattern into a list of files. Once it replaces the pattern with that list it is ready to run that "time" command....

Try:
time ksh -c "ls -1 *.abcdefg.Z"
# 7  
Old 07-10-2008
Results of the timing for ksh -c and find with maxdepth command
Perderabo,

Here are the results of timing the ksh -c and ls -1 command.

Code:
 time ksh -c "ls -1 *00abcdefg.Z"
ABC.00abcdefg.Z
AB.00abcdefg.Z


real    0m2.31s
user    0m0.87s
sys     0m1.03s

I tried the maxdepth option, however find complains of no such valid argument.

Code:
find: 0652-017 -maxdepth is not a valid option.

Thx
Jerardfjay
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to list and move files with spaces and wildcard?

I am writing a code that can move and archve all the files in a directory except the latest file based on file pattern provided in a controlfile. The filename is in the form of pattern. So basically we find the all the files of the pattern provided and archive all of them, leaving one latest file.... (3 Replies)
Discussion started by: Saanvi1
3 Replies

2. Shell Programming and Scripting

Parse through ~21,000 Database DDL statements -- Fastest way to perform search, replace and insert

Hello All: We are looking to search through 2000 files with around 21,000 statements where we have to search, replace and insert a pattern based on the following: 1) Parse through the file and check for CREATE MULTISET TABLE or CREATE SET TABLE statements.....and they always end with ON... (5 Replies)
Discussion started by: madhunk
5 Replies

3. Shell Programming and Scripting

How to copy all the contents of a list of files present in a folder to a particular file?

Hi All, I want to copy all the contents of a list of files in a folder to a particular file. i am using following command: cat dir/* >> newFile.txtIt's not working. Could you please help? Thanks, Pranav (3 Replies)
Discussion started by: Pranav Bhasker
3 Replies

4. Shell Programming and Scripting

List of files in a folder inclusive subfolder

Hi, I need to list the names of existing files in a specific folder. I have written a script for that, but the problem is, it is also picking up name of a subfolder that is there in that folder. I need only the list of files and not that subfolder. How to go about that ? Can anyone plz help... (2 Replies)
Discussion started by: Subhasis
2 Replies

5. Shell Programming and Scripting

Check folder existence using wildcard

Hi I would like to know how I can check whether there is one or more folders in the current directory which begins with e.g. 2011-11. Initially I figured that this could easily be done simply by: if ; then ... However if there is more than one folder which begins with 2011-11 then it... (1 Reply)
Discussion started by: aknu
1 Replies

6. UNIX for Dummies Questions & Answers

how to search and list file with wildcard character

hi, I want to search all files in the current working direcotry and to print in comma (,) seperated output. But I have two patterns to search for. Files will be in ABC20100508.DAT format. Search should happen on the format (ABC????????.DAT) along with date(20100508). I can do a ls... (2 Replies)
Discussion started by: anandapani
2 Replies

7. UNIX for Dummies Questions & Answers

List certain file in a folder and make list

Hi, im having problem that frustate me today. there are list of file in a folder that i want to grab the folder /subject/items/ in this folder there are this file CREATE_SUBxxxx.xml UPDATE_SUBxxxx.xml DELETE_SUBxxxx.xml loginresponsexxxxx.xml core how can i grab all the file... (1 Reply)
Discussion started by: andrisetia
1 Replies

8. Shell Programming and Scripting

How list of files apart from *.class files in certain folder

Hi Friends How to list of all files in a particular directory structure, with out listing a particular file extn. I mean to say in a folder structure I have many files ..I want to list them with out *.class files. Please help Thanks Joy (3 Replies)
Discussion started by: itsjoy2u
3 Replies

9. UNIX for Dummies Questions & Answers

Basic Q: getting list of all files of type within folder & subfolders

A painfully rudimentary UNIX question for somebody. I've been puzzling over this for the last hour but can't find the right command. I'm simply trying to get a list of all files - and their full paths - within a folder & subfolders which have extension .php and .js. That's it! No amount of... (1 Reply)
Discussion started by: AtomicPenguin
1 Replies

10. UNIX for Dummies Questions & Answers

Find wildcard .shtml files in wildcard directories and removing them- How's it done?

I'm trying to figure out how to build a small shell script that will find old .shtml files in every /tgp/ directory on the server and delete them if they are older than 10 days... The structure of the paths are like this: /home/domains/www.domain2.com/tgp/ /home/domains/www.domain3.com/tgp/... (1 Reply)
Discussion started by: Neko
1 Replies
Login or Register to Ask a Question