Shell Programming and Scripting

I have 100 Scripts, each containing 10-15 SQL's in them.

Each Script run for 40 mins to 1 hour 30 mins.

In the event of Failure in any step, if i re-start the Script, it will start running from the beginning. Which is waste of time. So in order handle this, i made the script to run from the failed point or to run it from any SQL the user want, by passing an argumnet for the Script while running It from the linux server box.

For example: If the Script has failed at 8th Step, when i re-run it will run from 8th Step. But, if user wants to run it from 6th Step, he can give ./Script_name 6. This will make the Script to run from 6th Step.


But now, i want to make the user to control the Script by passing argumnets from a table in ORACLE database.

There should be one Row entry for each of the 100 Scripts in one Table.

Code:

SCRIPT_NAME	SQL_NO	STATUS
Script 1	        1	                C (for completed)
Script 2	        3                       E (for Error)

Everytime the Script runs, the Row associated for the Script Should get dynamically updated with the SQL number that just completed.

And if there is any failure on 6th Step, when its re-run, it should run from the 6th Step. But if the user wants run from 5th Step, he should go to table and update the table with SQL_NO as 5 and should re-run the Script. Then the Script should start executing from the SQL_NO 5.

Kindle help me with your valuable suggestions.


Thanks for your time and help.

It sounds like you are trying to duplicate what the oracle job scheduler already does for free. If a tool already exists, reinventing it makes for problems down the road.

You will want to consider a daemon running in the background as the oracle user. The process checks the table, say once a minute to look for jobs to start.

But. If a data problem caused the job to fail in the first place, it will continue failing.
Unless a human intervenes to fix the issue.
So, you will need a table with date stamped status flags to indicate:

Code:

Successful completion - completed all jobs in one stream
Forced completion - mark this job stream as complete even though it failed
Failure pending restart - Got fixed please restart, Mr. job controller <- this one does what you asked
Failure - bombed, needs user attention - has sent an email
New job -  added every day/hour/week(?) by cron job to allow daily processing

Status should be indexed by some kind of unique job identifier, so that all New jobs when complete relates to one of the completion status steps. This is how the controller knows what to do. Plus, you can see repeated failures for the same job stream.

You need to add in the steps to the status 'Please restart me', so the daemon knows exactly what to do. Same is true for job properties like user/password/parameters

You also need something to notify users/support when a job bombs.

If you do not know enough to write a daemon, consider a set of shell scripts that are under the control of one master script which runs from crontab every minute or so. You can use the at or batch facility to do the submission.

If you are talking shell scripts, then you could have this structure:-

Code:

step=$1                         # Read step as first parameter
step="${step:=1}"               # Default to step one

until [ $step -gt 99 ]
do
 case $step in
   1) function_1 ;;
   2) function_2 ;;
   3) function_3 ;;
   4) function_4 ;;
 esac
 ((step=$step+1))
done

An alternate might be to set step and have an if ... then ... fi round each chunk of code like this:-

Code:

step=$1                         # Read step as first parameter
step="${step:=1}"               # Default to step one

if [ $step -le 1 ]
then
  function_1
fi

if [ $step -le 2 ]
then
  function_2
fi
:
:

They would sort of get over your request, but honestly this might just end up in a huge script trying to do too much in one go. You would be better to have a single script for each task you need to perform and keep it simple. Even better, write a utility script that takes an SQL deck as an argument and runs the appropriate code. You can then use a proper scheduler to set up dependencies, change the sequence of your code or whatever far more flexibly.

Of course, there is a cost to a proper scheduler and you would likely have to justify it. There are many out there on the market from those massively over-engineered but can run work on (just about) any platform to those that run on the local host only (although with a bit of ssh or rsh knowledge you can run processing elsewhere too)

At worst, I would at least suggest having a directory holding the scripts you want to run. At the start of the schedule, copy in each script, making sure each starts with a sequence number so that they naturally sort in the required order so, for example:-

Code:

01_01_01_script_to_setup_something
01_02_01_doing_this_bit_next
01_03_01_and_then_this_thing

This would allow you to insert an ad-hoc job if you need to, say 01_02_05_extra-bit_this_time

You then need a script to simply loop over the contents of the directory and (re)move files when the script completes error free. You would, of course have to document a recovery procedure if there was a failure and enable whomever to copy back an completed steps that needed to be re-run, say after a data restore and adjustment.

Your script to run the schedule could be pretty simple:-

Code:

for file in $dir/*
do
   $file
   RC=$?
   if [ $RC -eq 0 ]
   then
      mv $dir/$file $archive_dir/$file
      logger "Scheduled job complete: $file"
   else
      echo "Exciting $file with return code $RC"
      logger "Scheduled job failure: $file"
   fi
done

That said, it is a poor substitute for a proper scheduler. How complex a batch schedule do you think this might grow to? There could really be many ways to do this, so you really need to be certain of your logic of how it would need to flow, allow for exceptions or ad-hoc processes, recovery steps, restart intervention, data corrections, etc. etc.

Overall, I'd really suggest a proper 3rd party scheduler. Search for schedulers from CA, BMC, UC4, Axway, or a myriad of other software companies out there. They generally all allow you to manipulate the batch, restart from a specific point, bypass steps, run tasks in parallel and various other things. There may even be a decent freeware one that I'm not aware of - perhaps sourceforge or similar have built something.

Some are pretty, some are text only and very basic, but it depends what you need and what you can justify.


Robin

Thanks for your response.. But i am intended in creating such Script.. To read the input from Oracle table and start from the seq no based on the input obtained from table.

---------- Post updated at 05:08 AM ---------- Previous update was at 05:03 AM ----------

Thank you @rbatte1



The case statement which you have mentioned looks feasible. But, if we default every input to 1, won't the the script run from the beginning??


if i give the input as 6, i the script should run starting only from the 6th Step till the end.

Please give your thoughts on this.

Regards,
KIRAN

You would (re)start by running the script like this:-

Code:

$ my_script             # Run from the top, i.e. default is step 1
$ my_script 6           # Run from step 6 (or next step if 6 is not defined)

The default setting line means that if $step is not set then it to 1, so if it is already set, it retains that value. Write a little test script to try it out.

You could even have a step zero defined that is never run unless you force it.

Overall though, I would strongly suggest that you split it up and have one script for one task, then use a proper scheduler.




I hope that this helps,
Robin

Thanks Robin. Will try it out.

This is to make my requirement more clear


For Example:

Let's say I have 5 queries in my Script

Now, in the beginning of the Script, I will read an input from oracle table, which gives the SeQ_NO to execute.

Now when I get the seq number, I want to say in the Script to execute all the statements starting from the seq_no obtained.
(if the Seq_no obtained is 2, I should say in the Script to start from statement 3. If the input from the Table is NULL, then the Script should run from the beginning.)

My main requirement is that, I want to write command/ piece of code in the Script, which says start from statement number(based on what input we get from table) and execute till the end.


Please help me with your thoughts.