Run the Script from any stage by updating data from Oracle Table.
I have 100 Scripts, each containing 10-15 SQL's in them.
Each Script run for 40 mins to 1 hour 30 mins.
In the event of Failure in any step, if i re-start the Script, it will start running from the beginning. Which is waste of time. So in order handle this, i made the script to run from the failed point or to run it from any SQL the user want, by passing an argumnet for the Script while running It from the linux server box.
For example: If the Script has failed at 8th Step, when i re-run it will run from 8th Step. But, if user wants to run it from 6th Step, he can give ./Script_name 6. This will make the Script to run from 6th Step.
But now, i want to make the user to control the Script by passing argumnets from a table in ORACLE database.
There should be one Row entry for each of the 100 Scripts in one Table.
Everytime the Script runs, the Row associated for the Script Should get dynamically updated with the SQL number that just completed.
And if there is any failure on 6th Step, when its re-run, it should run from the 6th Step. But if the user wants run from 5th Step, he should go to table and update the table with SQL_NO as 5 and should re-run the Script. Then the Script should start executing from the SQL_NO 5.
Kindle help me with your valuable suggestions.
Thanks for your time and help.
Last edited by rbatte1; 01-25-2017 at 10:52 AM..
Reason: Added CODE tags for table display
It sounds like you are trying to duplicate what the oracle job scheduler already does for free. If a tool already exists, reinventing it makes for problems down the road.
You will want to consider a daemon running in the background as the oracle user. The process checks the table, say once a minute to look for jobs to start.
But. If a data problem caused the job to fail in the first place, it will continue failing.
Unless a human intervenes to fix the issue.
So, you will need a table with date stamped status flags to indicate:
Status should be indexed by some kind of unique job identifier, so that all New jobs when complete relates to one of the completion status steps. This is how the controller knows what to do. Plus, you can see repeated failures for the same job stream.
You need to add in the steps to the status 'Please restart me', so the daemon knows exactly what to do. Same is true for job properties like user/password/parameters
You also need something to notify users/support when a job bombs.
If you do not know enough to write a daemon, consider a set of shell scripts that are under the control of one master script which runs from crontab every minute or so. You can use the at or batch facility to do the submission.
This User Gave Thanks to jim mcnamara For This Post:
If you are talking shell scripts, then you could have this structure:-
An alternate might be to set step and have an if ... then ... fi round each chunk of code like this:-
They would sort of get over your request, but honestly this might just end up in a huge script trying to do too much in one go. You would be better to have a single script for each task you need to perform and keep it simple. Even better, write a utility script that takes an SQL deck as an argument and runs the appropriate code. You can then use a proper scheduler to set up dependencies, change the sequence of your code or whatever far more flexibly.
Of course, there is a cost to a proper scheduler and you would likely have to justify it. There are many out there on the market from those massively over-engineered but can run work on (just about) any platform to those that run on the local host only (although with a bit of ssh or rsh knowledge you can run processing elsewhere too)
At worst, I would at least suggest having a directory holding the scripts you want to run. At the start of the schedule, copy in each script, making sure each starts with a sequence number so that they naturally sort in the required order so, for example:-
This would allow you to insert an ad-hoc job if you need to, say 01_02_05_extra-bit_this_time
You then need a script to simply loop over the contents of the directory and (re)move files when the script completes error free. You would, of course have to document a recovery procedure if there was a failure and enable whomever to copy back an completed steps that needed to be re-run, say after a data restore and adjustment.
Your script to run the schedule could be pretty simple:-
That said, it is a poor substitute for a proper scheduler. How complex a batch schedule do you think this might grow to? There could really be many ways to do this, so you really need to be certain of your logic of how it would need to flow, allow for exceptions or ad-hoc processes, recovery steps, restart intervention, data corrections, etc. etc.
Overall, I'd really suggest a proper 3rd party scheduler. Search for schedulers from CA, BMC, UC4, Axway, or a myriad of other software companies out there. They generally all allow you to manipulate the batch, restart from a specific point, bypass steps, run tasks in parallel and various other things. There may even be a decent freeware one that I'm not aware of - perhaps sourceforge or similar have built something.
Some are pretty, some are text only and very basic, but it depends what you need and what you can justify.
Thanks for your response.. But i am intended in creating such Script.. To read the input from Oracle table and start from the seq no based on the input obtained from table.
---------- Post updated at 05:08 AM ---------- Previous update was at 05:03 AM ----------
Thank you @rbatte1
The case statement which you have mentioned looks feasible. But, if we default every input to 1, won't the the script run from the beginning??
if i give the input as 6, i the script should run starting only from the 6th Step till the end.
You would (re)start by running the script like this:-
The default setting line means that if $step is not set then it to 1, so if it is already set, it retains that value. Write a little test script to try it out.
You could even have a step zero defined that is never run unless you force it.
Overall though, I would strongly suggest that you split it up and have one script for one task, then use a proper scheduler.
I hope that this helps,
Robin
Last edited by rbatte1; 01-26-2017 at 07:56 AM..
Reason: Closing good wishes
Now, in the beginning of the Script, I will read an input from oracle table, which gives the SeQ_NO to execute.
Now when I get the seq number, I want to say in the Script to execute all the statements starting from the seq_no obtained.
(if the Seq_no obtained is 2, I should say in the Script to start from statement 3. If the input from the Table is NULL, then the Script should run from the beginning.)
My main requirement is that, I want to write command/ piece of code in the Script, which says start from statement number(based on what input we get from table) and execute till the end.
Please help me with your thoughts.
Last edited by rbatte1; 01-31-2017 at 12:47 PM..
Reason: Converted textual list to formatted numbered list and corrected spellings
Hi,
I am working on data stage 8.7 version and I want a script a to view the all log information of the data stage job from UNIX environment.
Can you please help me out by give the script.
Thanks in advance... (7 Replies)
Hi,
I want to write a shell script which will export data from oracle table .
I don't want to save that data . I want the queries .
Right now i am right clicking on the table and clicking on export as to my desktop .
Please let me know if any one have any idea . (2 Replies)
Hi,
i need to send emails to the mail ids i receive using the sql query .
SELECT L.ALARM_DEF,L.CRITERIA,L.DISTANCE_METRE,L.EMAIL,L.LAC_ID,L.LAST_CHECK,L.RANGE,
L.RESOURCE_MSISDN,LM.ADDRESS,LM.NAME FROM LANDMARK_ALARM_CONF l, LANDMARK lm
WHERE L.LANDMARK_ID=LM.LANDMARK_ID AND... (0 Replies)
Hi
I need to execute a select statement in a solaris environment with oracle database. The select statement returns number of rows of data.
I need the data to be inserted into a CSV file with proper format. For that we normally use "You have to select all your columns as one big string,... (2 Replies)
Hi,
I have a case where i am connecting to the oracle DB and reading a column value....
Script is in production...it was running fine,,,but suddenly now some times its started giving pain....
Script runs dailly....but sometimes its reading data from Oracle DB and sometimes its not rading the... (2 Replies)
Hi all,
I have a data like,
0,R001,2,D
this wants to be loaded into a oracle database table.
Pl let me know how this has to be done.
Thanks in advance (2 Replies)
Hi all,
How to load variable value into Oracle table? I have created a file validation shell script. After the validation completes, i need to update a Oracle table with the variable value...
Can someone help me how to do it? I have searched, but not able to get answer.
i have 4 variables... (2 Replies)
Hi all,
How to load variable value into Oracle table? I have created a file validation shell script. After the validation completes, i need to update a Oracle table with the variable value...
Can someone help me how to do it? I have searched, but not able to get answer.
i have 4... (1 Reply)
Hi,
Data file path (.txt)
Control file(.ctl)
I have delimited file(|).
Sample data:
1|name|50009|DS24|0|12
2|name|30009|DS24|0|13
3|name|20409|DS24|0|14
4|name|20009|DS24|0|15
5|name|10009|DS24|0|16
I want to load this data into a oracle table (update and insert)
Please help me... (1 Reply)
Hi friends,
i am having a variable declared in .profile.i am changing its value in a shell script and then i am connecting to oracle and then from there i am calling a .sql called update.sql
STATUS is the variable declared in the .profile
===============================
if
sqlplus <<END... (3 Replies)