Shell Programming and Scripting

Guys,

I wrote a script to do the following -

Initially, I was told that there would only be few files in a 15 minute time span and I cron-ed this script to run for every 5 mins and it was fast enough to load it in few minutes. Now, they are sending huge files and on couple occasions, the script ran longer than 10 minutes.

Now, Im running into an issue that I cant seem to grapple with on multiple levels.

I am sure a great many of you might have ran into this and want to learn as to what could be the best practice here. I am thinking of implementing a thread based approach but I never wrote thread programming in POSIX.

Please advise

I presume that you loading script has something like:-

Code:

for file in *
do
   sqlload connection $file to table options
done

If you have your data that is to be loaded in a directory /data then when you script runs, it could:-In this way, you can have the script running multiple times but they will load separate data files. Of course, you still have to consider:-You could also just slow dow the frequency of your cron job I suppose......


I hope that this logic might help.
Robin

rbatte1, thanks a bunch for taking time and writing back to me on this. Let me try my way around this here...

Let us know how you get on or if I have just caused more confusion.

All questions welcome and someone here should be able to help.



Regards,
Robin