Required help on a Generic File-watcher script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Required help on a Generic File-watcher script
# 1  
Old 03-09-2016
Required help on a Generic File-watcher script

Hi All,

Good morning...

I have prepared a shell script which will monitor files in a certain folder and if available, SCP it to a destination path.

Now the challenge I'm facing is my script is currently SCP-ing to only a single destination path. Wherever different destination path is in the scenario, my script is failing.

I'm using 2 properties files.
1. Files to Monitor
2. File-watcher configuration (contains server name, incoming path, destination path, scp user name etc etc)

The script is as below:
Code:
#!/bin/ksh

### This is for reading File Watcher config file and SCP the files to certain server
# >> Author: Saps
# >> Dated: 09-FEB-2016
# >> Version: 0.1
#

##Variables
CONFIG_PATH=/path/to/config/file
CONFIG_FILE=gen.FileWatcher.config
FILES_TO_MONITOR=app.outFile.FileWatcher.properties
CURR_HOST_NM=`uname -n`
 . $CONFIG_PATH/$CONFIG_FILE 2>/dev/null
sleepAppTime=$appt
sleepFWTime=$fwt
sleepSCPTime=$scpt
PROD_HOST=$p1h
UAT_HOST=$u1h
SCP_USER_NM=$scpuser
DEST_PATH=$dfpath
SCP_HOST=$scph
compFilePath=$ifpath
vDate=`date +%d%m%y%H%M`
# endFileSize=0

###### FUNCTION CREATION ################
function fileWatcherConcept
{
  echo "[INFO - `date`] Starting file watcher process..."
  sleep $sleepFWTime
  while true
  do
    filePath=$ofpath
    echo "[INFO - `date`] File path is: $filePath"
    for fileName in $(cat $CONFIG_PATH/$FILES_TO_MONITOR)
    do
      echo "[INFO - `date`] Watching file: $fileName"
      if [ ! -s $filePath/$fileName ]
      then
        echo "[INFO - `date`] The file: $filePath/$fileName is not arrived yet... will check after $sleepFWTime secs."
        sleep $sleepFWTime
      else
        echo "[INFO - `date`] File arrival is in progress..."
        sleep $sleepAppTime
        startFileSize=`ls -l $filePath/$fileName|awk '{print $5}'`
        echo "[DEBUG - `date`] File Size of $fileName on arrival: $startFileSize"
        echo "[INFO - `date`] SCP not completed yet... will check after $sleepSCPTime secs. "
        sleep $sleepSCPTime
        endFileSize=`ls -l $filePath/$fileName|awk '{print $5}'`
        echo "[DEBUG - `date`] File Size of $fileName after $sleepSCPTime secs: $endFileSize"
        if [ $startFileSize -eq $endFileSize ]
        then
          echo "[INFO - `date`] << FILE SIZE >> matches: $endFileSize is final size."
          echo "[INFO - `date`] The file: $filePath/$fileName has arrived... "
          echo "[INFO - `date`] Initializing file transfer via SCP process..."
          # SCP script / command goes here. also used scp errorCode is 0 or not
          echo "[INFO - `date`] SCP Command in execution is: /usr/local/bin/scp -B -q $filePath/$fileName $SCP_USER_NM@$SCP_HOST:$DEST_PATH/$fileName"
          /usr/local/bin/scp -B -q $filePath/$fileName $SCP_USER_NM@$SCP_HOST:$DEST_PATH/$fileName
          # scpErrCode=0 #<-- for debug
          scpErrCode=$?
          if [ $scpErrCode -ne 0 ]
          then
            echo "[DEBUG - `date`] Error in SCP. Error Code: $scpErrCode"
          else
            echo "[INFO - `date`] SCP completed successfully..."
            echo "[INFO - `date`] Sending SCP Completion Flag..." 
            echo "[INFO - `date`] SCP Command in execution for .comp file is: /usr/local/bin/scp -B -q $compFilePath/$fileName.comp $SCP_USER_NM@$SCP_HOST:$DEST_PATH/$fileName.comp"
            /usr/local/bin/scp -B -q $compFilePath/$fileName.comp $SCP_USER_NM@$SCP_HOST:$DEST_PATH/$fileName.comp
            sleep $sleepAppTime
            # mv command to move files from out to archive dir with timestamp
            mv $filePath/$fileName $filePath/$fileName.ARCHIVE
            echo "[INFO - `date`] file has moved to archive folder."
          fi
        else
          continue;
        fi
        sleep $sleepFWTime
      fi
    done
  done
}


#### MAIN ######

if [ "$CURR_HOST_NM" != "$PROD_HOST" ]
then
  echo "[INFO - `date`] Current host is $CURR_HOST_NM"
  echo "[INFO - `date`] Loading UAT configuration..."
  ## function goes here
  # echo "function call for UAT" #<-- for debug purpose
  sleep $sleepAppTime
  fileWatcherConcept
elif [ "$CURR_HOST_NM" != "$UAT_HOST" ]
then
  echo "[INFO - `date`] Current host is $CURR_HOST_NM"
  echo "[INFO - `date`] Loading PROD configuration..."
  ## function goes here
  # echo "function call for PROD" #<-- for debug purpose
  sleep $sleepAppTime
  fileWatcherConcept
else
  echo "[ERROR - `date`] Either the server: $CURR_HOST_NM or the configuration is incorrect. Exiting..."
  exit 1
fi

# END of FW Script

The contents of gen.FileWatcher.config:
Code:
# This is file watcher config file for application out-directory
#

::server::
u1h=uat.server
p1h=prod.server

::scpHost::
scpuh=uat.server
scpph=prod.server

::userIDs::
scpuser=scpUserNM

::appIDs:: #<-- this area just added, if any concept can be implemented using application IDs
app1=xyz
app2=abc

::filesPath::
ofpath=/outgoing/file/path/out/dir
ifpath=/incoming/file/path/in/dir
dfpath=/scp/destination/file/path/xyz_out
# dfpath1=/scp/destination/file/path/xyz_out #<-- if any concept can be implemented
# dfpath2=/scp/destination/file/path/abc_out #<-- if any concept can be implemented

::sleepTimes::
appt=3
fwt=5
scpt=10

The contents of app.outFile.FileWatcher.properties:
Code:
file_name_1.DAT
file_name_2.DAT
file_name_3.DAT
file_name_4.DAT

All kind of suggestion is welcome to achieve this.

Please let me know how can "SCP to different file path" can be implemented.

Thanks in advance,
Saps.
# 2  
Old 03-12-2016
I would approach the problem differently.
If you have control of the sending process then, have the sender put the data file in the temp directory, then put a job file in the requests directory.
The job file contains the instructions for what to do with the data file.
The job file extension is used to create separate job streams, in our instance priority.
This entire process is run as a cron job with whatever frequency you choose. We found that a long running process is too fragile, and you don't find out is not running or is hung up on something until an angry user arrives.
If you cannot have two files sent, then use 'fuser' to determine if the file is still in the process of being received, and if it is still in use, bypass it until the next iteration.

Code:
#!/bin/ksh                                                                      
PATH=/bin:/usr/bin:/usr/lib                                                     
BATCH=$1                                                                        
proc=$$                                                                         
if [ -r /u/email/$BATCH.pid ]                                                   
then                                                                            
        echo Previous job `cat /u/email/$BATCH.pid` is still running            
        exit                                                                    
fi                                                                              
echo $proc >/u/email/$BATCH.pid                                                 
cd /u/email                                                                     
list=`ls requests|grep $BATCH`                                                  
if [ "$list" != "" ]                                                            
        then                                                                    
        for request in $list                                                    
        do                                                                      
        fuser $request                                                          
        if [ $? != 0 ]                                                          
          then                                                                  
          req=`echo $request|cut -d"." -f1`                                     
          export req                                                            
          IFS="~"                                                               
          #cat requests/$request |read user email from subject memo attachments 
          cat requests/$request | read  user destination                        
          
          IFS=" "                                                               
          cd /u/email/temp                                                      
          #php emailstub.php $user $email "$subject" $memo $attachments         
          #or in your instance scp $req* $user@destination:/incoming            
          cd /u/email                                                           
          mv /u/email/temp/$req* done                                           
          mv /u/email/requests/$request done                                    
        fi                                                                      
        sleep 10                                                                
        done                                                                    
fi                                                                              
rm /u/email/$BATCH.pid

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Generic script to load file details(ls -ltr) in to a database.

All, I am trying to create a report on the duration of an ETL load from the file arrival to the final dump in to a database for SLA's. Does anyone have any guidance or ideas on how metadata can be extracted; information of a file: like file name, created timestamp, count of records and load... (1 Reply)
Discussion started by: pradeepp
1 Replies

2. Shell Programming and Scripting

File watcher script

if test -n "$(find/data/path/ 'filename.txst' -print-quit) then echo "file found" exit 0 else echo "file not found" exit 46 fi So I basically looking to understand the if test -n "$(find/data/path/ 'filename.txst' -print-quit) line. Pls help to elaborate what this command... (6 Replies)
Discussion started by: Nsharma3006
6 Replies

3. UNIX for Dummies Questions & Answers

Watcher Script

Hi Guys, I hope anyone can help me on how to create a watcher-script. The script will act as watcher wherein whenever it will find an error (say the word Error) in a file, it will prompt the user. Please help! br, rymnd_12345 (3 Replies)
Discussion started by: rymnd_12345
3 Replies

4. Shell Programming and Scripting

Help needed with File Watcher script

Hi all, I am writing a script to do the following: 1) Check if there are any new <csv> files (in abc directory) to process 2) If there is new file, then perform some operation on that file. 3) Move it to a different location(after step2 is successful) 4) Do further processing on the file ... (2 Replies)
Discussion started by: stunnerz_84
2 Replies

5. Shell Programming and Scripting

File Watcher Help !!

Hi Experts I will have be having 3 types of files in directory file1_p0_date file1_p1_date file1_p2_date As soon as it sees any of the files it needs to kick off another process and also would need the file name For this I am creating a file watcher script which will look for file1* My... (2 Replies)
Discussion started by: westernmichnov
2 Replies

6. UNIX for Advanced & Expert Users

Need Script for File Watcher

Hi, I need a script for "FILE WATCHER" I have in a remote FTP location server named MAIN _104 EveryDay mrng 8 i hav to search couple of folders(COL, ATT, TRA) in that FTP server (MAIN_104), and if any files are present in that folders i hav to bring back to my computer and store it anywhere... (4 Replies)
Discussion started by: nani1984
4 Replies

7. Shell Programming and Scripting

ksh script to create a generic csv file from different source formats

Hi all, I have a requirement to create a "superset" file out of a number of different sources with some different and some same columns. We intend to have a manually updateable SuperSetCols.csv which would look like "ColA","ColB","ColC","ColD","ColE","ColF","ColG" so someday we may add... (3 Replies)
Discussion started by: Leedor
3 Replies

8. Shell Programming and Scripting

File Watcher

Hi Please help me in this I want to execute a shell script abc.ksh. But I only want it to execute if file XYZ is not present. If file XYZ is present than I want to unix to sleep for 5 Sec and than agaian check for XYX existence. if it sleeps for more than 30 seconds ( 6 time )I want it to... (3 Replies)
Discussion started by: er_zeeshan05
3 Replies

9. Shell Programming and Scripting

generic way - help required

All, This is what I did: Suppose: $ line="23|12|21" $ eval $(echo "$line" | awk -F'|' '{print "V1="$1";V2="$2";V3="$3}') $ echo $V1 23 $ echo $V2 12 $ echo $V3 21 (3 Replies)
Discussion started by: uwork72
3 Replies

10. Shell Programming and Scripting

Generic Shell Script to Archive a file

Would appreciate if any one can paste a generic schell script to archive a file with date stamp by passing the file with fullpath as parameter For Eg. /apps/scripts/Archive_File.sh /data_home/project_home/file.txt this should place the file in the following directory ... (8 Replies)
Discussion started by: mak1600
8 Replies
Login or Register to Ask a Question