Sponsored Content
Top Forums Shell Programming and Scripting Required help on a Generic File-watcher script Post 302968381 by saps19 on Wednesday 9th of March 2016 02:12:11 AM
Old 03-09-2016
Required help on a Generic File-watcher script

Hi All,

Good morning...

I have prepared a shell script which will monitor files in a certain folder and if available, SCP it to a destination path.

Now the challenge I'm facing is my script is currently SCP-ing to only a single destination path. Wherever different destination path is in the scenario, my script is failing.

I'm using 2 properties files.
1. Files to Monitor
2. File-watcher configuration (contains server name, incoming path, destination path, scp user name etc etc)

The script is as below:
Code:
#!/bin/ksh

### This is for reading File Watcher config file and SCP the files to certain server
# >> Author: Saps
# >> Dated: 09-FEB-2016
# >> Version: 0.1
#

##Variables
CONFIG_PATH=/path/to/config/file
CONFIG_FILE=gen.FileWatcher.config
FILES_TO_MONITOR=app.outFile.FileWatcher.properties
CURR_HOST_NM=`uname -n`
 . $CONFIG_PATH/$CONFIG_FILE 2>/dev/null
sleepAppTime=$appt
sleepFWTime=$fwt
sleepSCPTime=$scpt
PROD_HOST=$p1h
UAT_HOST=$u1h
SCP_USER_NM=$scpuser
DEST_PATH=$dfpath
SCP_HOST=$scph
compFilePath=$ifpath
vDate=`date +%d%m%y%H%M`
# endFileSize=0

###### FUNCTION CREATION ################
function fileWatcherConcept
{
  echo "[INFO - `date`] Starting file watcher process..."
  sleep $sleepFWTime
  while true
  do
    filePath=$ofpath
    echo "[INFO - `date`] File path is: $filePath"
    for fileName in $(cat $CONFIG_PATH/$FILES_TO_MONITOR)
    do
      echo "[INFO - `date`] Watching file: $fileName"
      if [ ! -s $filePath/$fileName ]
      then
        echo "[INFO - `date`] The file: $filePath/$fileName is not arrived yet... will check after $sleepFWTime secs."
        sleep $sleepFWTime
      else
        echo "[INFO - `date`] File arrival is in progress..."
        sleep $sleepAppTime
        startFileSize=`ls -l $filePath/$fileName|awk '{print $5}'`
        echo "[DEBUG - `date`] File Size of $fileName on arrival: $startFileSize"
        echo "[INFO - `date`] SCP not completed yet... will check after $sleepSCPTime secs. "
        sleep $sleepSCPTime
        endFileSize=`ls -l $filePath/$fileName|awk '{print $5}'`
        echo "[DEBUG - `date`] File Size of $fileName after $sleepSCPTime secs: $endFileSize"
        if [ $startFileSize -eq $endFileSize ]
        then
          echo "[INFO - `date`] << FILE SIZE >> matches: $endFileSize is final size."
          echo "[INFO - `date`] The file: $filePath/$fileName has arrived... "
          echo "[INFO - `date`] Initializing file transfer via SCP process..."
          # SCP script / command goes here. also used scp errorCode is 0 or not
          echo "[INFO - `date`] SCP Command in execution is: /usr/local/bin/scp -B -q $filePath/$fileName $SCP_USER_NM@$SCP_HOST:$DEST_PATH/$fileName"
          /usr/local/bin/scp -B -q $filePath/$fileName $SCP_USER_NM@$SCP_HOST:$DEST_PATH/$fileName
          # scpErrCode=0 #<-- for debug
          scpErrCode=$?
          if [ $scpErrCode -ne 0 ]
          then
            echo "[DEBUG - `date`] Error in SCP. Error Code: $scpErrCode"
          else
            echo "[INFO - `date`] SCP completed successfully..."
            echo "[INFO - `date`] Sending SCP Completion Flag..." 
            echo "[INFO - `date`] SCP Command in execution for .comp file is: /usr/local/bin/scp -B -q $compFilePath/$fileName.comp $SCP_USER_NM@$SCP_HOST:$DEST_PATH/$fileName.comp"
            /usr/local/bin/scp -B -q $compFilePath/$fileName.comp $SCP_USER_NM@$SCP_HOST:$DEST_PATH/$fileName.comp
            sleep $sleepAppTime
            # mv command to move files from out to archive dir with timestamp
            mv $filePath/$fileName $filePath/$fileName.ARCHIVE
            echo "[INFO - `date`] file has moved to archive folder."
          fi
        else
          continue;
        fi
        sleep $sleepFWTime
      fi
    done
  done
}


#### MAIN ######

if [ "$CURR_HOST_NM" != "$PROD_HOST" ]
then
  echo "[INFO - `date`] Current host is $CURR_HOST_NM"
  echo "[INFO - `date`] Loading UAT configuration..."
  ## function goes here
  # echo "function call for UAT" #<-- for debug purpose
  sleep $sleepAppTime
  fileWatcherConcept
elif [ "$CURR_HOST_NM" != "$UAT_HOST" ]
then
  echo "[INFO - `date`] Current host is $CURR_HOST_NM"
  echo "[INFO - `date`] Loading PROD configuration..."
  ## function goes here
  # echo "function call for PROD" #<-- for debug purpose
  sleep $sleepAppTime
  fileWatcherConcept
else
  echo "[ERROR - `date`] Either the server: $CURR_HOST_NM or the configuration is incorrect. Exiting..."
  exit 1
fi

# END of FW Script

The contents of gen.FileWatcher.config:
Code:
# This is file watcher config file for application out-directory
#

::server::
u1h=uat.server
p1h=prod.server

::scpHost::
scpuh=uat.server
scpph=prod.server

::userIDs::
scpuser=scpUserNM

::appIDs:: #<-- this area just added, if any concept can be implemented using application IDs
app1=xyz
app2=abc

::filesPath::
ofpath=/outgoing/file/path/out/dir
ifpath=/incoming/file/path/in/dir
dfpath=/scp/destination/file/path/xyz_out
# dfpath1=/scp/destination/file/path/xyz_out #<-- if any concept can be implemented
# dfpath2=/scp/destination/file/path/abc_out #<-- if any concept can be implemented

::sleepTimes::
appt=3
fwt=5
scpt=10

The contents of app.outFile.FileWatcher.properties:
Code:
file_name_1.DAT
file_name_2.DAT
file_name_3.DAT
file_name_4.DAT

All kind of suggestion is welcome to achieve this.

Please let me know how can "SCP to different file path" can be implemented.

Thanks in advance,
Saps.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Generic Shell Script to Archive a file

Would appreciate if any one can paste a generic schell script to archive a file with date stamp by passing the file with fullpath as parameter For Eg. /apps/scripts/Archive_File.sh /data_home/project_home/file.txt this should place the file in the following directory ... (8 Replies)
Discussion started by: mak1600
8 Replies

2. Shell Programming and Scripting

generic way - help required

All, This is what I did: Suppose: $ line="23|12|21" $ eval $(echo "$line" | awk -F'|' '{print "V1="$1";V2="$2";V3="$3}') $ echo $V1 23 $ echo $V2 12 $ echo $V3 21 (3 Replies)
Discussion started by: uwork72
3 Replies

3. Shell Programming and Scripting

File Watcher

Hi Please help me in this I want to execute a shell script abc.ksh. But I only want it to execute if file XYZ is not present. If file XYZ is present than I want to unix to sleep for 5 Sec and than agaian check for XYX existence. if it sleeps for more than 30 seconds ( 6 time )I want it to... (3 Replies)
Discussion started by: er_zeeshan05
3 Replies

4. Shell Programming and Scripting

ksh script to create a generic csv file from different source formats

Hi all, I have a requirement to create a "superset" file out of a number of different sources with some different and some same columns. We intend to have a manually updateable SuperSetCols.csv which would look like "ColA","ColB","ColC","ColD","ColE","ColF","ColG" so someday we may add... (3 Replies)
Discussion started by: Leedor
3 Replies

5. UNIX for Advanced & Expert Users

Need Script for File Watcher

Hi, I need a script for "FILE WATCHER" I have in a remote FTP location server named MAIN _104 EveryDay mrng 8 i hav to search couple of folders(COL, ATT, TRA) in that FTP server (MAIN_104), and if any files are present in that folders i hav to bring back to my computer and store it anywhere... (4 Replies)
Discussion started by: nani1984
4 Replies

6. Shell Programming and Scripting

File Watcher Help !!

Hi Experts I will have be having 3 types of files in directory file1_p0_date file1_p1_date file1_p2_date As soon as it sees any of the files it needs to kick off another process and also would need the file name For this I am creating a file watcher script which will look for file1* My... (2 Replies)
Discussion started by: westernmichnov
2 Replies

7. Shell Programming and Scripting

Help needed with File Watcher script

Hi all, I am writing a script to do the following: 1) Check if there are any new <csv> files (in abc directory) to process 2) If there is new file, then perform some operation on that file. 3) Move it to a different location(after step2 is successful) 4) Do further processing on the file ... (2 Replies)
Discussion started by: stunnerz_84
2 Replies

8. UNIX for Dummies Questions & Answers

Watcher Script

Hi Guys, I hope anyone can help me on how to create a watcher-script. The script will act as watcher wherein whenever it will find an error (say the word Error) in a file, it will prompt the user. Please help! br, rymnd_12345 (3 Replies)
Discussion started by: rymnd_12345
3 Replies

9. Shell Programming and Scripting

File watcher script

if test -n "$(find/data/path/ 'filename.txst' -print-quit) then echo "file found" exit 0 else echo "file not found" exit 46 fi So I basically looking to understand the if test -n "$(find/data/path/ 'filename.txst' -print-quit) line. Pls help to elaborate what this command... (6 Replies)
Discussion started by: Nsharma3006
6 Replies

10. UNIX for Beginners Questions & Answers

Generic script to load file details(ls -ltr) in to a database.

All, I am trying to create a report on the duration of an ETL load from the file arrival to the final dump in to a database for SLA's. Does anyone have any guidance or ideas on how metadata can be extracted; information of a file: like file name, created timestamp, count of records and load... (1 Reply)
Discussion started by: pradeepp
1 Replies
All times are GMT -4. The time now is 06:47 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy