Recursive folder search faster than find?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Recursive folder search faster than find?
# 1  
Old 05-15-2015
Recursive folder search faster than find?

I'm trying to find folders created by a propritary data aquisition software with the .aps ending--yes, I have never encountered folder with a suffix before (some files also end in .aps) and sort them by date. I need the whole path

ls -dt "$dataDir"*".aps"does exactly what I want except for the recursion. find "$dataDir" -type d -name '*.aps' does everyting but is excruciatingly slow on windows remote shares. It takes about 45 seconds to find 27 folders. 8 are one subdirectory below $dataDir and one is 2 subdirectories lower. There are a few hundred files and each .aps folder has 7 subfolders with a bunch of files (many called setup.aps which seems to be slowing the find). I know I can get use the printf inside find to get a date column and sort and cut it but it would be even slower.

ls -dt does everything I want except the recursion. the -R option in ls is a completely different animal and does not do the kind of recursion I'm looking for. -1R is sometimes useful but in the above command it does nothing.

Last edited by Michael Stora; 05-15-2015 at 03:10 PM..
# 2  
Old 05-15-2015
You are kinda stuck. The handling of your request lives on the other side. The problem is - what happens on the windows side? Changing commands on UNIX is not going to help much.

I am assuming you are using CIFS/samba, try: Faster Samba (SMB / CIFS) Share Performance - Eggplant Systems and Design
# 3  
Old 05-15-2015
I'm working under Cygwin64 in a all-Windows corporate environment. (This utility runs side by side with the data aquisition software allowing users to edit run names, comments, validate/invalidate/exclude data. The propritary software is so user unfriendly and clunky (I guess that's what the thousands of dollars is for Smilie ). I happen to have special privilages on the share drive I am using for development but won't for the deployed system.

ls is plenty fast but I can't do the recursion. Not sure what other commands are out there. On the plus side, I have some really fast hardware to run on Smilie

Mike

---------- Post updated at 11:32 AM ---------- Previous update was at 11:14 AM ----------

part of it is windows file sharing. It takes about 0.6 seconds vs 45 seconds on my local machine. However, find itself is dozens of times slower than ls.

Mike

---------- Post updated at 11:45 AM ---------- Previous update was at 11:32 AM ----------

I'm living with find find "$dataDir" -type d -name '*.aps' -printf '%T@|%p\n' | sort -nr | cut -d"|" -f 2 ("|"being an windows disallowed path character) until one of the forum Awk or Perl deities pops in Smilie It is taking no longer than the find itself (~45 seconds).

Mike
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

A faster way to read and search

I have a simple script that reads in data from fileA.txt and searches line by line for that data in multiple files (*multfiles.txt). It only prints the data when there is more than 1 instance of it. The problem is that its really slow (3+ hours) to complete the entire process. There are nearly 1500... (10 Replies)
Discussion started by: ncwxpanther
10 Replies

2. OS X (Apple)

Search recursive

before posting, I have tried to find my answer elsewhere. no luck. I need to find a file buried in a folder somewhere. Master folder has 10 sub folders. each sub folder has folders too. I found this but it does nothing I am on Mac and use Applescript. do shell script "find... (2 Replies)
Discussion started by: sbrady
2 Replies

3. UNIX for Dummies Questions & Answers

Help needed - find command for recursive search

Hi All I have a requirement to find the file that are most latest to be modified in each directory. Can somebody help with the command please? E.g of the problem. The directory A is having sub directory which are having subdirectory an so on. I need a command which will find the... (2 Replies)
Discussion started by: sudeep.id
2 Replies

4. Shell Programming and Scripting

Faster search needed

Hope you guys out there can help. I have 2 files as below: file 1: 111,222,333,444,555,666 777,888,999,000,111,222 111,222,333,444,555,888 file 2: 666,AAA 222,BBB 888,CCC I want to get the 6th column from file 1 (example, 666) and check in file 2 for the value in the 2nd column... (9 Replies)
Discussion started by: daytripper1021
9 Replies

5. Shell Programming and Scripting

recursive search and copy

Hello again. Well, I need help again sooner as I thought. Now I want to search for files with a known name within all subdirs, and copy the to differently named files in the same directory. For example if I had only one file to copy, I would just usecp fileName newFileNamebut to do this... (1 Reply)
Discussion started by: cabaciucia
1 Replies

6. UNIX for Advanced & Expert Users

Recursive search for filenames containing ..

Hi all, Since my gopher server doesn't like filenames containing 2 or more consecutive dots in a filename, I'd like to do a search for them and replace them with... well, let's say underscores... I've tried a oneliner or 2 from other posts, but they don't seem to work well with locating dots.... (3 Replies)
Discussion started by: Evert
3 Replies

7. UNIX for Advanced & Expert Users

Recursive directory search using ls instead of find

I was working on a shell script and found that the find command took too long, especially when I had to execute it multiple times. After some thought and research I came up with two functions. fileScan() filescan will cd into a directory and perform any operations you would like from within... (8 Replies)
Discussion started by: newreverie
8 Replies

8. UNIX for Dummies Questions & Answers

recursive search and ftp

Could someone help me in recursive search and ftp'ing the files to remote server? The host machine will have /dir1/dira/list_of_files1 /dir1/dirb/list_of_files2 /dir1/dirc/list_of_files3 . . . so., I need to search from dir1 recursively (only one level down) and find all the files that... (1 Reply)
Discussion started by: brahmi
1 Replies

9. UNIX for Dummies Questions & Answers

Zip recursive content of folder when (not current directory=

Hi, Is there a way to zip the content (recursively) of a folder other then the current directory, and keep the directory structure intact? Example: /var/tmp/myfolder ----------------- file1 ----------------- file2 ----------------- folder1 ------------------------ file3 Now I want... (3 Replies)
Discussion started by: jimih
3 Replies

10. UNIX for Advanced & Expert Users

find file with date and recursive search for a text

Hey Guyz I have a requirement something like this.. a part of file name, date of modification of that file and a text is entered as input. like Date : 080206 (MMDDYY format.) filename : hotel_rates text : Jim now the file hotel_rates.ZZZ.123 (creation date is Aug 02 2006) should be... (10 Replies)
Discussion started by: rosh0623
10 Replies
Login or Register to Ask a Question