The UNIX and Linux Forums  

Go Back   The UNIX and Linux Forums > Top Forums > UNIX for Dummies Questions & Answers
Google UNIX.COM


UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
pattern matching mercuryshipzz Shell Programming and Scripting 4 01-14-2008 08:01 PM
AWK pattern matching, first and last smb_uk Shell Programming and Scripting 10 12-27-2007 06:03 PM
pattern matching malle Shell Programming and Scripting 3 01-31-2007 02:23 AM
pattern matching larryase UNIX for Dummies Questions & Answers 3 11-22-2004 03:54 PM
Pattern Matching danhodges99 UNIX for Dummies Questions & Answers 2 02-27-2003 12:03 PM

Reply
 
Submit Tools LinkBack Thread Tools Search this Thread Display Modes
  #1  
Old 01-21-2008
Registered User
 

Join Date: Jan 2008
Posts: 7
Slow pattern matching ls or rm in a large directory

Hello! I'm new here so hopefully I'm posting in the right forum.

Got a storage that is NFS mounted to a Sun box running Solaris 8.
When I run a pattern matching ls or rm, it will take as long as 1-2mins to
complete. Patterns are usually consist of prefix and a *, for example...
"ls -l abc123*" or "rm -f abc123*"

Pattern matching ls or rm will be nearly instant the 2nd time I run it, but
1-2mins the 1st time.

The directory is huge. Contains some 100k to 500k small 20k sized files.

I get really fast response if I ls or rm with the exact file name.

Is this normal?

I was told it's a bandwidth issue but I could transfer a large 180MB file
to the NFS storage in 10secs.

Any help much appreciated.
Reply With Quote
Forum Sponsor
  #2  
Old 01-21-2008
sysgate's Avatar
Unix based
 

Join Date: Nov 2006
Location: /root
Posts: 1,200
This is normal, it has to do with the number of files within the folder. Use "find" + "xargs" to limit the number of output; I remember that we had such discussion a while ago here, let me see the search option
P.S. I can't find anything useful, but having in mind that this is NFS mounted folder would add even more latency, given the RPC calls.

Last edited by sysgate; 01-21-2008 at 07:06 AM.
Reply With Quote
  #3  
Old 01-21-2008
Registered User
 

Join Date: Jan 2008
Posts: 7
Quote:
Originally Posted by sysgate View Post
This is normal, it has to do with the number of files within the folder. Use "find" + "xargs" to limit the number of output; I remember that we had such discussion a while ago here, let me see the search option
P.S. I can't find anything useful, but having in mind that this is NFS mounted folder would add even more latency, given the RPC calls.
The output is no more than 2-5 lines though... as only 2-5 files will match.
Reply With Quote
  #4  
Old 01-21-2008
reborg's Avatar
Administrator
 

Join Date: Mar 2005
Location: Ireland
Posts: 3,642
Quote:
Originally Posted by JT-KGY View Post
The directory is huge. Contains some 100k to 500k small 20k sized files.
That's the problem, for ls the issue is not the size of files, but the size of the directory. I am not going to explain at length why this happens, but Perderabo has previously done so a number of times, for example here.
Reply With Quote
  #5  
Old 01-21-2008
Registered User
 

Join Date: Jan 2008
Posts: 7
Quote:
Originally Posted by reborg View Post
That's the problem, for ls the issue is not the size of files, but the size of the directory. I am not going to explain at length why this happens, but Perderabo has previously done so a number of times, for example here.
I understand the operation will be slow.. but 2-5mins slow?
On a local disk, it will take about 1-3 sec to complete the call.

I understand that NFS mounted storage will be slower but from 1-3 secs to
2-5 mins?
Reply With Quote
  #6  
Old 01-21-2008
Registered User
 

Join Date: Jan 2008
Posts: 7
Quote:
Originally Posted by reborg View Post
Perderabo has previously done so a number of times, for example here.
I read Perderabo's post.. but still...

When I search for a specific file in the directory... I get instant response.
It's when I search for files using a wild card that I get very very poor
performance...

(Say file abc12345.html vs abc123*.html)
Reply With Quote
  #7  
Old 01-22-2008
sysgate's Avatar
Unix based
 

Join Date: Nov 2006
Location: /root
Posts: 1,200
The issue is with the wildcard itself, take a look at glob(), consider 100 000 system glob() calls, add NFS latency, RPC calls, slow network, overloaded server, and you'll get the answer, whereas strict file match is a different thing. "Is the file there - yes / no" - pretty simple.
Reply With Quote
Google The UNIX and Linux Forums
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes




All times are GMT -7. The time now is 11:34 PM.


Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited.
The UNIX and Linux Forums Content Copyright ©1993-2008. All Rights Reserved.Ad Management by RedTyger Visit The Complex Event Processing Blog

Content Relevant URLs by vBSEO 3.2.0