help on most efficient search


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting help on most efficient search
# 8  
Old 04-05-2008
Question

Quote:
Originally Posted by prvnrk
Hello,

We have a directory with 15 sub-directories where each sub-directory contains 1.5 to 2 lakhs of files in it. Daily, around 300-500 files will be uploaded to each sub-directory.

Now, i need to get the list of files received today in most efficient way. I tried using "find with newer option" and also the "ls -ltr with tail" but both are taking long time to provide the list of images received today.

Please advise me on the most efficient way (should take least time possible) to find today's files.


TIA
Prvn
Please explain what do you mean by "lakhs" Smilie
# 9  
Old 04-06-2008
Sorry for my regional language.

lakh (or lac) means 100 thousand (100,000).


Thanks
Prvn
# 10  
Old 04-06-2008
shamrock: that's why I put in the wiki link anyway ...

prvnrk: it's probably not so simple as to just compile the thing on Solaris, because it depends on the availability of various system calls and other pieces of infrastructure. The one I was thinking of is related to dbus but I can't recall its name. Here's another one: Monitor Linux file system events with inotify

Googling for "solaris dnotify" mainly brings up links discussing how it's not available, but I didn't look very closely; maybe you can find something similar. Actually, "solaris inotify" brings up some rather promising hits; look at Summer of Code - Genunix and UNIX man pages : inotify (7) which says the following:

Quote:
The inotify API is Linux specific. Some other systems provide similar mechanisms, e.g., FreeBSD has kqueue, and Solaris has /dev/poll.
Edit: Hmm, I guess /dev/poll is not what you want. The Summer of Code link takes you to a thread where they discuss a possible future mechanism ("future" relative to 2006; we can only speculate what happened then ...) so things are not so promising. One of the messages says you have the basic kernel support but it should be exposed to userland. Here's another one for you: Nabble - Gnome - Lib - Gamin - General - Try to port gamin to Solaris

Last edited by era; 04-06-2008 at 02:33 AM.. Reason: Maybe not so hopeful after all
# 11  
Old 04-06-2008
Quote:
Originally Posted by prvnrk
Sorry for my regional language.

lakh (or lac) means 100 thousand (100,000).


Thanks
Prvn
I had this problem of using lakhs instead of "K" or "M" - that turned out to be a real fun for others in a meeting. This happened few years ago when I was into this industry for the first time.

We need to use international attributes all the time. Smilie

I dont mean to offend anybody just thought of sharing this.
# 12  
Old 04-06-2008
We can kind of tackle this problem.

There are 2 approaches to do this.

1) Use a book keeping file which maintains a list of filenames and its status, whether its processed or not which would indicate indirectly whether its a new file or not

2) much more easier way use a table and populate the table accordingly with the filename and the status.

Both are same but based on the number of files you are handling and the need for book keeping you need to make a call for the option
# 13  
Old 04-08-2008
Thank you Era.

I tried dnotify, inotify and FAM on Solaris but could not setup successfully. All these are working just great on Linux.

As I could not get any info on using /dev/poll on Solaris, i did not try it out yet.


Thanks
Prvn
# 14  
Old 04-08-2008
Could you hack UFS to provide a suitable hook directly? Sounds tacky but it might not even be all that hard (but if this is an important production system, management approval might be a tough issue).
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Efficient way to search array in text file by awk

I have one array SPLNO with approx 10k numbers.Now i want to search the subscriber number from MDN.TXT file (containing approx 1.5 lac record)from the array.if subscriber number found in array it will perform below operation.my issue is that it's taking more time because for one number it's search... (6 Replies)
Discussion started by: siramitsharma
6 Replies

2. Shell Programming and Scripting

Search and replace multiple patterns in a particular column only - efficient script

Hi Bigshots, I have a pattern file with two columns. I have another data file. If column 1 in the pattern file appears as the 4th column in the data file, I need to replace it (4th column of data file) with column 2 of the pattern file. If the pattern is found in any other column, it should not... (6 Replies)
Discussion started by: ss112233
6 Replies

3. Shell Programming and Scripting

Need an efficient way to search for a tag in an xml file having millions of rows

Hi, I have an XML file with around 1 billion rows in it and i am trying to find the number of times a particular tag occurs in it. The solution i am using works but takes a lot of time (~1 hr) .Please help me with an efficient way to do this. Lets say the input file is <Root> ... (13 Replies)
Discussion started by: Sheel
13 Replies

4. Shell Programming and Scripting

Help with Efficient Looping

Hello guys My requirement is to read a file with parent-child relationship we need to iterate through each row to find its latest child. for eg. parent child ABC PQR PQR DEF DEF XYZ Expected Output ABC XYZ PQR XYZ DEF XYZ Script Logic : read parent from file seach child... (4 Replies)
Discussion started by: joshiamit
4 Replies

5. Shell Programming and Scripting

Better and efficient way to reverse search a file for first matched line number.

How to reverse search for a matched string in a file. Get line# of the first matched line. I am getting '2' into 'lineNum' variable. But it feels like I am using too many commands. Is there a better more efficiant way to do this on Unix? abc.log aaaaaaaaaaaaa bbbbbbbbbbbbb... (11 Replies)
Discussion started by: kchinnam
11 Replies

6. UNIX for Dummies Questions & Answers

Is this regex efficient?

I want to match the red portion: 9784323456787-Unknown Phrase with punctuation "Some other PhrASE." Is this the best regex to match this? '978\{10\}-*' (4 Replies)
Discussion started by: glev2005
4 Replies

7. UNIX for Advanced & Expert Users

efficient repace

some of the data i receive has been typed in manually due to which there are often places where i find 8 instead of ( and the incorrect use of case what according to you is the best way to correct such data. The data has around 20,000 records. The value i want to change is in the 4th field.... (2 Replies)
Discussion started by: VGR
2 Replies

8. Shell Programming and Scripting

efficient search

Hi, i have 2 files each with 200K lines. Each line contains a number. Now, i need to get the list of numbers existing in one fine and NOT in other file. I'm doing this by reading each number from 1 file and grepping on other file. But this taking LOT of time. Is there any efficient way of doing... (14 Replies)
Discussion started by: prvnrk
14 Replies

9. Shell Programming and Scripting

Is there a more efficient way?

I'm using korn shell to connect to oracle, retrieve certain values, put them in a list, and iterate through them. While this method works, I can't help but think there is an easier method. If you know of one, please suggest a shorter, more efficient method. ############### FUNCTIONS ... (6 Replies)
Discussion started by: SelectSplat
6 Replies

10. UNIX for Advanced & Expert Users

Efficient Dispatching

Does anyone know what's new with Efficient dispatching in the Solaris 2.8 release (vs Solaris 2.6) release? Specifically, does anyone know of a good website to get detailed information on thread dispatching using efficient dispatching in solaris 2.8? Thank you. (1 Reply)
Discussion started by: uchachra
1 Replies
Login or Register to Ask a Question