Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Generic Filewatcher

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Generic Filewatcher
# 1  
Old 03-06-2013
Generic Filewatcher
Hi,

I have a requirement wherein i need to have a generic file watcher in place.

On presence of a file in a particular directory,the file watcher should identify the related config file and execute the series of the shell scripts mentioned in the config file.

eg.Config file
Code:
a.sh
b.sh c.sh d.sh
e.sh

In this case,we can have
a.sh executed first followed sequentially
by b.sh,c.sh,d.sh which can be executed in parallel.post b.sh,c.sh and d.sh execution successfully,
e.sh needs to be executed.
Also for each shell script their stderr and stdout should be logged.

So overall,the file watcher might watch for any file existence in a particular directory again configured in a master file.
eg.contents of the master file
Code:
ABC.txt ABC.config
PQR.txt PQR.config

ABC.config might be as below:
Code:
a.sh
b.sh c.sh d.sh
e.sh

PQR.config might be as below:
Code:
q.sh
r.sh 
s.sh

Would it be possible to achieve this using basic bash shell scripting?

Regards,
Dikesh Shah.
Last edited by jim mcnamara; 04-14-2013 at 08:23 PM..
# 2  
Old 03-06-2013
What do a.sh, b.sh, c.sh, d.sh, e.sh do?

What should happen to the input file after the scripts complete?

These config files sound quite similar to makefiles, or could be made so.
This User Gave Thanks to Corona688 For This Post:
dikesm
# 3  
Old 03-06-2013
You could try something like this, not they after finding a watch file you should probably delete or move it aside to stop actions repeating on every check:

Code:
function load_conf
{
  POS=0
  while read line
  do
     S[POS]="$line"
     let POS=POS+1
  done < $1
}

while read WATCH CONF
do
    POS=${#WF[@]}
    WF[POS]=$WATCH
    CF[POS]=$CONF
done < master

for((i=0;i<${#WF[@]};i++)) {
    if [ -f "${WF[i]}" ]
    then
        load_conf ${CF[i]}
        for((j=0;j<${#S[@]};j++)) {
            echo "${S[j]}" | while read -a run
            do
                for((k=0;k<${#run[@]};k++)) {
                    eval ${run[k]} \> $(basename ${run[k]}).out "2>" $(basename ${run[k]}).stderr &
                }
                wait
            done
        }
    fi
}

This User Gave Thanks to Chubler_XL For This Post:
dikesm
# 4  
Old 04-13-2013
Quote:
Originally Posted by Corona688
What do a.sh, b.sh, c.sh, d.sh, e.sh do?

What should happen to the input file after the scripts complete?

These config files sound quite similar to makefiles, or could be made so.
The input file is a file containing some data to be processed.The input file would be archived after processing.

The scripts might do simple stuffs like checking whether the source file name is correct as per logic,or check whether there aint any duplication of records in the files,logic to do a data quality check.

These individual scripts run on their own and export some shell variables along with 0 return code in case of success and non zero return code for any failures.

Regards,
Dikesh Shah.

---------- Post updated at 04:50 AM ---------- Previous update was at 04:47 AM ----------

Quote:
Originally Posted by Chubler_XL
You could try something like this, not they after finding a watch file you should probably delete or move it aside to stop actions repeating on every check:

Code:
function load_conf
{
  POS=0
  while read line
  do
     S[POS]="$line"
     let POS=POS+1
  done < $1
}

while read WATCH CONF
do
    POS=${#WF[@]}
    WF[POS]=$WATCH
    CF[POS]=$CONF
done < master

for((i=0;i<${#WF[@]};i++)) {
    if [ -f "${WF[i]}" ]
    then
        load_conf ${CF[i]}
        for((j=0;j<${#S[@]};j++)) {
            echo "${S[j]}" | while read -a run
            do
                for((k=0;k<${#run[@]};k++)) {
                    eval ${run[k]} \> $(basename ${run[k]}).out "2>" $(basename ${run[k]}).stderr &
                }
                wait
            done
        }
    fi
}


Can you please explain what does the above code does?
I am a bit new to complex shell programming.
# 5  
Old 04-14-2013
Code:
function load_conf
{
  POS=0
  while read line
  do
     S[POS]="$line"
     let POS=POS+1
  done < $1
}

This function takes 1 argument that is the name of a config file. It then reads this file into the array S[]. For you 1st example config file S would contain:
Code:
    S[0]="a.sh"
    S[1]="b.sh c.sh d.sh"
    S[2]="e.sh"


Code:
while read WATCH CONF
do
    POS=${#WF[@]}
    WF[POS]=$WATCH
    CF[POS]=$CONF
done < master

This loads the master file into WF[] (watch files) and CF[] (config files). For you example master file:
Code:
    WF[0]="ABC.txt"
    WF[1]="PQR.txt"
    CF[0]="ABC.config"
    CF[1]="PQE.config"

Code:
for((i=0;i<${#WF[@]};i++)) {

Process the WF array 1 element at a time

Code:
    if [ -f "${WF[i]}" ]
    then
        load_conf ${CF[i]}

if the watch file exists then load the config file for this watch file.

Code:
for((j=0;j<${#S[@]};j++)) {
    echo "${S[j]}" | while read -a run
    do

for each S[] element loaded above read the commands into the run[] array

Code:
for((k=0;k<${#run[@]};k++)) {
    eval ${run[k]} \> $(basename ${run[k]}).out "2>" $(basename ${run[k]}).stderr &
}
wait

run the seperate commands from the S[] array in the background redirecting the std out and stderr to .out and .stderr files.
Then wait for each of these background processes to finish before starting on the next S[] entry.
# 6  
Old 04-14-2013
Attached are the testing files I used from your description, the test scripts basically just echo their names to stdout and stderr and sleep for random amounts of time.
dikesm.tar (20.0 KB) 
# 7  
Old 04-14-2013
Quote:
Originally Posted by Chubler_XL
Code:
    eval ${run[k]} \> $(basename ${run[k]}).out "2>" $(basename ${run[k]}).stderr &

Why use eval?

Regards,
Alister
This User Gave Thanks to alister For This Post:
Chubler_XL
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Filewatcher

hi All, I ned to write a filewatcher script, with following requirements. 1. The script should look for the file in every 5 min. 2. If the file is found, it should check in every 3 min the size of file. 3. if the file size has not changed in last 4 iterations (i.e. in last 12 min), the... (2 Replies)
Discussion started by: alok2082
2 Replies

2. UNIX for Dummies Questions & Answers

Filewatcher job

Hi Friends iam using a filewatcher job which checks the path in intervals below is the script #!/bin/ksh fileflag=0 timer1=0 check_interval=120 # check every 2 minutes (( check_interval_minutes=${check_interval}/60 )) while do if then echo "My file exists now..." | mailx -s... (7 Replies)
Discussion started by: robertbrown624
7 Replies

3. Shell Programming and Scripting

Autosys filewatcher + ksh script

Hi, A ------> B ------> C I have a scenario where each day, my server B would ftp to server A and pull (A,B,C,D,E) from a specific directory. Server C would need files (B,D) only when server B finished receiving from server A. These files change everyday, so sometimes it takes longer... (3 Replies)
Discussion started by: arex876
3 Replies

4. UNIX for Advanced & Expert Users

Is aclocal.m4 generic?

can we copy higher version aclocal to our software. Is there any good book for automake,aclocal.m4,configure.sub,configure.guess that explains clearly about how they are related , how to modify them etc Thanks Gopi (1 Reply)
Discussion started by: Gopi Krishna P
1 Replies

5. Shell Programming and Scripting

FileWatcher Script

Hi All, Sorry to post these many questions on UNIX. i am new to unix & got only UNIX work in my organization. I need to make a script for File Arrival Check. 1. The script should wait for indicator file for configured amount of time. 2. If the file is not received after the configured... (4 Replies)
Discussion started by: Amit.Sagpariya
4 Replies

6. Shell Programming and Scripting

generic way - help required

All, This is what I did: Suppose: $ line="23|12|21" $ eval $(echo "$line" | awk -F'|' '{print "V1="$1";V2="$2";V3="$3}') $ echo $V1 23 $ echo $V2 12 $ echo $V3 21 (3 Replies)
Discussion started by: uwork72
3 Replies

7. Shell Programming and Scripting

Problem with filewatcher...

Hi everyone, Please find the script for Filewatcher rule file,which does the simple job of moving the files whenever it dectects to another directory.And whenever it detects the cmd_mm.stop file,it should terminate the job. INTERVAL 60 ON_FILEWATCH ${HLD}/CMD/* CREATE 0 1 2 0400 5 THEN... (2 Replies)
Discussion started by: bhagat.singh-j
2 Replies

8. UNIX for Dummies Questions & Answers

GENERIC Kernel

How do I update, change, reconfigure or whatever it is that I have to do, in order to rid the GENERIC label. It just means that it is the basic kernel shipped with the OS right? Im using FBSD 4.5 (2 Replies)
Discussion started by: savage
2 Replies
Login or Register to Ask a Question