The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Top Forums > UNIX for Dummies Questions & Answers
.
google unix.com



UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
pattern matching mercuryshipzz Shell Programming and Scripting 4 01-14-2008 11:01 PM
AWK pattern matching, first and last smb_uk Shell Programming and Scripting 10 12-27-2007 09:03 PM
pattern matching malle Shell Programming and Scripting 3 01-31-2007 05:23 AM
pattern matching larryase UNIX for Dummies Questions & Answers 3 11-22-2004 06:54 PM
Pattern Matching danhodges99 UNIX for Dummies Questions & Answers 2 02-27-2003 03:03 PM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 01-21-2008
JT-KGY JT-KGY is offline
Registered User
  
 

Join Date: Jan 2008
Posts: 7
Slow pattern matching ls or rm in a large directory

Hello! I'm new here so hopefully I'm posting in the right forum.

Got a storage that is NFS mounted to a Sun box running Solaris 8.
When I run a pattern matching ls or rm, it will take as long as 1-2mins to
complete. Patterns are usually consist of prefix and a *, for example...
"ls -l abc123*" or "rm -f abc123*"

Pattern matching ls or rm will be nearly instant the 2nd time I run it, but
1-2mins the 1st time.

The directory is huge. Contains some 100k to 500k small 20k sized files.

I get really fast response if I ls or rm with the exact file name.

Is this normal?

I was told it's a bandwidth issue but I could transfer a large 180MB file
to the NFS storage in 10secs.

Any help much appreciated.
  #2 (permalink)  
Old 01-21-2008
sysgate's Avatar
sysgate sysgate is offline Forum Advisor  
Unix based
  
 

Join Date: Nov 2006
Location: Bulgaria
Posts: 1,318
This is normal, it has to do with the number of files within the folder. Use "find" + "xargs" to limit the number of output; I remember that we had such discussion a while ago here, let me see the search option
P.S. I can't find anything useful, but having in mind that this is NFS mounted folder would add even more latency, given the RPC calls.

Last edited by sysgate; 01-21-2008 at 10:06 AM..
  #3 (permalink)  
Old 01-21-2008
JT-KGY JT-KGY is offline
Registered User
  
 

Join Date: Jan 2008
Posts: 7
Quote:
Originally Posted by sysgate View Post
This is normal, it has to do with the number of files within the folder. Use "find" + "xargs" to limit the number of output; I remember that we had such discussion a while ago here, let me see the search option
P.S. I can't find anything useful, but having in mind that this is NFS mounted folder would add even more latency, given the RPC calls.
The output is no more than 2-5 lines though... as only 2-5 files will match.
  #4 (permalink)  
Old 01-21-2008
reborg's Avatar
reborg reborg is offline Forum Staff  
Administrator
  
 

Join Date: Mar 2005
Location: Ireland
Posts: 4,209
Quote:
Originally Posted by JT-KGY View Post
The directory is huge. Contains some 100k to 500k small 20k sized files.
That's the problem, for ls the issue is not the size of files, but the size of the directory. I am not going to explain at length why this happens, but Perderabo has previously done so a number of times, for example here.
  #5 (permalink)  
Old 01-21-2008
JT-KGY JT-KGY is offline
Registered User
  
 

Join Date: Jan 2008
Posts: 7
Quote:
Originally Posted by reborg View Post
That's the problem, for ls the issue is not the size of files, but the size of the directory. I am not going to explain at length why this happens, but Perderabo has previously done so a number of times, for example here.
I understand the operation will be slow.. but 2-5mins slow?
On a local disk, it will take about 1-3 sec to complete the call.

I understand that NFS mounted storage will be slower but from 1-3 secs to
2-5 mins?
  #6 (permalink)  
Old 01-21-2008
JT-KGY JT-KGY is offline
Registered User
  
 

Join Date: Jan 2008
Posts: 7
Quote:
Originally Posted by reborg View Post
Perderabo has previously done so a number of times, for example here.
I read Perderabo's post.. but still...

When I search for a specific file in the directory... I get instant response.
It's when I search for files using a wild card that I get very very poor
performance...

(Say file abc12345.html vs abc123*.html)
  #7 (permalink)  
Old 01-22-2008
sysgate's Avatar
sysgate sysgate is offline Forum Advisor  
Unix based
  
 

Join Date: Nov 2006
Location: Bulgaria
Posts: 1,318
The issue is with the wildcard itself, take a look at glob(), consider 100 000 system glob() calls, add NFS latency, RPC calls, slow network, overloaded server, and you'll get the answer, whereas strict file match is a different thing. "Is the file there - yes / no" - pretty simple.
Closed Thread

Bookmarks

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On




All times are GMT -4. The time now is 12:47 AM.


Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0