Shell script for scheduling

chanduhyd4u · #1 (**permalink**) 01-21-2005

Hi guys,

I am new guy for unix. I need help writing the following shell script.
My requirement is:

I have few pl/sql procedures(lets say 10). I want to run some of them (say 5) parallally in the background and once these are completed trigger the next ones one after other.

Could some one give some ideas on how to keep track of the first 5 background processes and how it can be implemented.(some sample script would be appreciated.)

Thanks
Chandu

locustfurnace · #2 (**permalink**) 01-22-2005

Welcome, this seems to be a homework question. You should post what you have worked out already. If this is a homework question, please read the Unix.com forum's Rules.

Quote:

Originally Posted by http://www.unix.com/showthread.php?t=2971

(6) Do not post classroom or homework problems.

google · #3 (**permalink**) 01-22-2005

This may be a legitimate post. Create a temp table in your database, have each procedure post back to the table. This is how our project tends to do these sorts of things. Table is small and has columns such as: job ID, current status (in progress, success, failure) and a date/time column of the last update.Drop the table if it is not needed after your run.

chanduhyd4u · #4 (**permalink**) 01-22-2005

This is actually our project requirement. I described here it as a small assignment, but actually we have about 150 procedures and some of them can be run parallally and some of them have dependency. Our objective is to write a shell script to schedule this task. I have no idea about how to start with and request your ideas/suggestions.

Google: could you be more specific. When the first set of procedures triggered in the background how do we monitor for its status and trigger the next dependent job when it is done. I mean do we need to use some kind script which keeps on checking this job status table in specific intervals?

thanks for your suggestions in advance.

google · #5 (**permalink**) 01-22-2005

We have a couple of tables set up:a job dependency table (holds job id and jod dependency ID among other things) and aother is a job master table (table containing all jobs and job ids) and finally a job status table. Shell scripts control the execution of stored procedures. Before a procedure is executed the shell runs a small sql statement to check the job status table to see if the current job is able to run. If the last status is success, and the dependency is also met everything is good. The controlling shell also updates the job status table based upon the status of the PL/SQL it kicked off (in progress, success, failure). this works well for us but we dont have 150 procedures to run in parallel either

google · #6 (**permalink**) 01-22-2005

You may not need this but your first post mentions you are new to UNIX, disregard if you already know this. Here is a link that shows how you execute a stored procedure from a shell and capture output variables.

tmarikle · #7 (**permalink**) 01-28-2005

Quote:

Originally Posted by chanduhyd4u

This is actually our project requirement. I described here it as a small assignment, but actually we have about 150 procedures and some of them can be run parallally and some of them have dependency. Our objective is to write a shell script to schedule this task. I have no idea about how to start with and request your ideas/suggestions.

What a complicated request! You are trying to code a work flow system in a shell script.

If you're new to UNIX, you will certainly have trouble wrapping your arms around someone else's scripts.

Conceptually, here is what you want to do:

Your first goal is to build the framework for interacting with Oracle. I find that communicating to Oracle with a sqlplus coprocess to be very easy to control your various interactions.

Code:

sqlplus -s /nolog |&

Then you send messages to the coprocess.

Code:

print -p "connect un/pw@db"
print -p "prompt 'sql complete'"

Then you can read back Oracle's responses simillarly as follows:

Code:

while read -p LINE
do
    # This is necessary so that the loop doesn't freeze waiting for more output
    if print $LINE | grep 'sql complete'
    then
        break
    fi
    # More result analysis...
    if ...
done

This works for PL/SQL code wrappers, dynamic SQL, etc. In fact, the only thing that isn't so great is if you have 1000s of lines of results and you want to process them with the while read command.

The next thing that you need is a mechanism to drive your execution priorities. This can be done with one or more tables and defining some basic rules. The simplest will be the processing order.

You'll also want to identify the don't care procedures that can run in the background. This is more tricky but can be done fairly easy. I did this for my current project where I need to bulk load data using sqlldr and where I can't use direct path loads (this works at blazingly fast speeds; my tests showed nearly one million records per minute in one test case). I split the task into five threads and each thread loads a different portion of the file. You can sychronize information by wrapping the command in a pair of braces which causes the code to execute in a subshell. The direct the output to a common file for analyzing the results.

Code:

{
    sqlldr un/pw@db ...
    # put any kind of stuff inside the braces that will help you digest the processing results
    print result code=$?
} >> some_output_file_common_to_all_threads &

# save the process ID
mypidlist="$mypidlist $!"
...
# later on, when you want to synchronize the threads you have to wait on the background pids.
# The following wait is more reliable than the unix wait (for me that is  :( )
while [ $(ps -p $(print ${mypidlist} |
          tr -d ' ' |
          nawk '{print substr($0,1,length($0)-1)}') |
          wc -l) -gt 1 ]
do
   sleep 2
done

# now process information related to your thread's output
while read LINE
do
    ...
done < some_output_file_common_to_all_threads

Your serially executed procedures/tasks run in the coprocessor and your "don't care procedures" can be run in their own sqlplus sessions independent of the coprocessor.

Coprocess exacmple:

Code:

print -p "SET SERVEROUTPUT ON"
print -p "BEGIN"
print -p "    call_my_proc;"
print -p "END;"
print -p "/"
print -p "PROMPT 'sql complete'"

while read -p LINE
do
   ... same as before
done

Background example:

Code:

{
sqlplus -s /nolog <<EOF
connect us/pw@db
SET SERVEROUTPUT ON
BEGIN
     call_my_proc;
     call_another_one_if_you_like;
     DBMS_OUTPUT.PUT_LINE ('whatever message you want');
END;
/
EOF
} >> some_other_file_as_before &
mypids="$mypids $!"

If you really want to monitor status of the background procedures, use the facilities with Oracle for long oeprations and then use your coprocessor to periodically query the information. Or you can use your own table, whatever you want but the Oracle longops facility is designed for this simple kind of job communication.

This can also serve as a mechanism for synchronizing your background tasks.

This is a very simplistic description for solving your very complex requirement but I use this kind of technique now with great success.

Thomas

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Passing arguments to a shell script from file while scheduling in cron	weblogicsupport	SUN Solaris	4	01-27-2008 11:16 PM
Problem with scheduling a shell script on cygwin using cron	shash	UNIX for Dummies Questions & Answers	4	08-09-2007 06:08 PM
Scheduling the execution of a shell script	sumesh.abraham	Shell Programming and Scripting	1	12-06-2006 07:25 AM
script scheduling	GNMIKE	Shell Programming and Scripting	2	07-11-2005 01:39 AM
scheduling script to run another script on Solaris	andrei	Shell Programming and Scripting	3	03-27-2005 04:25 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread: