Shell Programming and Scripting

#! /usr/bin/bash
/path/to/job1 &
/path/to/job2 &
/path/to/job3 &
wait
/path/to/job4 &
/path/to/job5 &
/path/to/job6 &
wait
exit 0

join/to/job 1 % ...
pico
wait
exit 0,1

Hi:
Could you explain what does this extra code does.
-------
join/to/job 1 % ...
pico
wait
exit 0,1
-------
Thanks,
Chandu

What a complicated request! You are trying to code a work flow system in a shell script.

If you're new to UNIX, you will certainly have trouble wrapping your arms around someone else's scripts.

Conceptually, here is what you want to do:

Your first goal is to build the framework for interacting with Oracle. I find that communicating to Oracle with a sqlplus coprocess to be very easy to control your various interactions.

Code:

sqlplus -s /nolog |&

Then you send messages to the coprocess.

Code:

print -p "connect un/pw@db"
print -p "prompt 'sql complete'"

Then you can read back Oracle's responses simillarly as follows:

Code:

while read -p LINE
do
    # This is necessary so that the loop doesn't freeze waiting for more output
    if print $LINE | grep 'sql complete'
    then
        break
    fi
    # More result analysis...
    if ...
done

This works for PL/SQL code wrappers, dynamic SQL, etc. In fact, the only thing that isn't so great is if you have 1000s of lines of results and you want to process them with the while read command.

The next thing that you need is a mechanism to drive your execution priorities. This can be done with one or more tables and defining some basic rules. The simplest will be the processing order.

You'll also want to identify the don't care procedures that can run in the background. This is more tricky but can be done fairly easy. I did this for my current project where I need to bulk load data using sqlldr and where I can't use direct path loads (this works at blazingly fast speeds; my tests showed nearly one million records per minute in one test case). I split the task into five threads and each thread loads a different portion of the file. You can sychronize information by wrapping the command in a pair of braces which causes the code to execute in a subshell. The direct the output to a common file for analyzing the results.

Code:

{
    sqlldr un/pw@db ...
    # put any kind of stuff inside the braces that will help you digest the processing results
    print result code=$?
} >> some_output_file_common_to_all_threads &

# save the process ID
mypidlist="$mypidlist $!"
...
# later on, when you want to synchronize the threads you have to wait on the background pids.
# The following wait is more reliable than the unix wait (for me that is  :( )
while [ $(ps -p $(print ${mypidlist} |
          tr -d ' ' |
          nawk '{print substr($0,1,length($0)-1)}') |
          wc -l) -gt 1 ]
do
   sleep 2
done

# now process information related to your thread's output
while read LINE
do
    ...
done < some_output_file_common_to_all_threads

Your serially executed procedures/tasks run in the coprocessor and your "don't care procedures" can be run in their own sqlplus sessions independent of the coprocessor.

Coprocess exacmple:

Code:

print -p "SET SERVEROUTPUT ON"
print -p "BEGIN"
print -p "    call_my_proc;"
print -p "END;"
print -p "/"
print -p "PROMPT 'sql complete'"

while read -p LINE
do
   ... same as before
done

Background example:

Code:

{
sqlplus -s /nolog <<EOF
connect us/pw@db
SET SERVEROUTPUT ON
BEGIN
     call_my_proc;
     call_another_one_if_you_like;
     DBMS_OUTPUT.PUT_LINE ('whatever message you want');
END;
/
EOF
} >> some_other_file_as_before &
mypids="$mypids $!"

If you really want to monitor status of the background procedures, use the facilities with Oracle for long oeprations and then use your coprocessor to periodically query the information. Or you can use your own table, whatever you want but the Oracle longops facility is designed for this simple kind of job communication.

This can also serve as a mechanism for synchronizing your background tasks.

This is a very simplistic description for solving your very complex requirement but I use this kind of technique now with great success.

Thomas

I have an SFTP coprocess whose output is redirected to a file (as shown in the script below). Now how can I read the output of the coprocess:

exec 4>$FTPLOG
sftp nakarapu@serverx >&4 2>&4 |&
print -p mget $filename
print -p !echo "The FTP is complete"
read -p LINE
echo $LINE
print -p bye
wait

What changes to the above code can make it read the LINE from the coprocess.

(Btw, this is an attempt to synchronize with the coprocess. If you have a better way to know when the coprocess finishes the ftp, please let me know.)


Thanks in advance
Naveen Akarapu

I have never been able to get ssh, scp, or sftp to work via a coprocess due to how it forks a child process. You need to use expect or Perl expect; that's what I have used.

T

expect is not installed in our systems.

But I am guessing there must be some kind of I/O indirection that allows me to read the output of the coprocess.

Code:

read -p LINE

That would normally do it but, as I said, secure shell commands fork a child process and that I/O can't be captured by the coprocess. Sorry.

t