I have prepared a shell script which will monitor files in a certain folder and if available, SCP it to a destination path.
Now the challenge I'm facing is my script is currently SCP-ing to only a single destination path. Wherever different destination path is in the scenario, my script is failing.
I'm using 2 properties files.
1. Files to Monitor
2. File-watcher configuration (contains server name, incoming path, destination path, scp user name etc etc)
The script is as below:
Code:
#!/bin/ksh
### This is for reading File Watcher config file and SCP the files to certain server
# >> Author: Saps
# >> Dated: 09-FEB-2016
# >> Version: 0.1
#
##Variables
CONFIG_PATH=/path/to/config/file
CONFIG_FILE=gen.FileWatcher.config
FILES_TO_MONITOR=app.outFile.FileWatcher.properties
CURR_HOST_NM=`uname -n`
. $CONFIG_PATH/$CONFIG_FILE 2>/dev/null
sleepAppTime=$appt
sleepFWTime=$fwt
sleepSCPTime=$scpt
PROD_HOST=$p1h
UAT_HOST=$u1h
SCP_USER_NM=$scpuser
DEST_PATH=$dfpath
SCP_HOST=$scph
compFilePath=$ifpath
vDate=`date +%d%m%y%H%M`
# endFileSize=0
###### FUNCTION CREATION ################
function fileWatcherConcept
{
echo "[INFO - `date`] Starting file watcher process..."
sleep $sleepFWTime
while true
do
filePath=$ofpath
echo "[INFO - `date`] File path is: $filePath"
for fileName in $(cat $CONFIG_PATH/$FILES_TO_MONITOR)
do
echo "[INFO - `date`] Watching file: $fileName"
if [ ! -s $filePath/$fileName ]
then
echo "[INFO - `date`] The file: $filePath/$fileName is not arrived yet... will check after $sleepFWTime secs."
sleep $sleepFWTime
else
echo "[INFO - `date`] File arrival is in progress..."
sleep $sleepAppTime
startFileSize=`ls -l $filePath/$fileName|awk '{print $5}'`
echo "[DEBUG - `date`] File Size of $fileName on arrival: $startFileSize"
echo "[INFO - `date`] SCP not completed yet... will check after $sleepSCPTime secs. "
sleep $sleepSCPTime
endFileSize=`ls -l $filePath/$fileName|awk '{print $5}'`
echo "[DEBUG - `date`] File Size of $fileName after $sleepSCPTime secs: $endFileSize"
if [ $startFileSize -eq $endFileSize ]
then
echo "[INFO - `date`] << FILE SIZE >> matches: $endFileSize is final size."
echo "[INFO - `date`] The file: $filePath/$fileName has arrived... "
echo "[INFO - `date`] Initializing file transfer via SCP process..."
# SCP script / command goes here. also used scp errorCode is 0 or not
echo "[INFO - `date`] SCP Command in execution is: /usr/local/bin/scp -B -q $filePath/$fileName $SCP_USER_NM@$SCP_HOST:$DEST_PATH/$fileName"
/usr/local/bin/scp -B -q $filePath/$fileName $SCP_USER_NM@$SCP_HOST:$DEST_PATH/$fileName
# scpErrCode=0 #<-- for debug
scpErrCode=$?
if [ $scpErrCode -ne 0 ]
then
echo "[DEBUG - `date`] Error in SCP. Error Code: $scpErrCode"
else
echo "[INFO - `date`] SCP completed successfully..."
echo "[INFO - `date`] Sending SCP Completion Flag..."
echo "[INFO - `date`] SCP Command in execution for .comp file is: /usr/local/bin/scp -B -q $compFilePath/$fileName.comp $SCP_USER_NM@$SCP_HOST:$DEST_PATH/$fileName.comp"
/usr/local/bin/scp -B -q $compFilePath/$fileName.comp $SCP_USER_NM@$SCP_HOST:$DEST_PATH/$fileName.comp
sleep $sleepAppTime
# mv command to move files from out to archive dir with timestamp
mv $filePath/$fileName $filePath/$fileName.ARCHIVE
echo "[INFO - `date`] file has moved to archive folder."
fi
else
continue;
fi
sleep $sleepFWTime
fi
done
done
}
#### MAIN ######
if [ "$CURR_HOST_NM" != "$PROD_HOST" ]
then
echo "[INFO - `date`] Current host is $CURR_HOST_NM"
echo "[INFO - `date`] Loading UAT configuration..."
## function goes here
# echo "function call for UAT" #<-- for debug purpose
sleep $sleepAppTime
fileWatcherConcept
elif [ "$CURR_HOST_NM" != "$UAT_HOST" ]
then
echo "[INFO - `date`] Current host is $CURR_HOST_NM"
echo "[INFO - `date`] Loading PROD configuration..."
## function goes here
# echo "function call for PROD" #<-- for debug purpose
sleep $sleepAppTime
fileWatcherConcept
else
echo "[ERROR - `date`] Either the server: $CURR_HOST_NM or the configuration is incorrect. Exiting..."
exit 1
fi
# END of FW Script
The contents of gen.FileWatcher.config:
Code:
# This is file watcher config file for application out-directory
#
::server::
u1h=uat.server
p1h=prod.server
::scpHost::
scpuh=uat.server
scpph=prod.server
::userIDs::
scpuser=scpUserNM
::appIDs:: #<-- this area just added, if any concept can be implemented using application IDs
app1=xyz
app2=abc
::filesPath::
ofpath=/outgoing/file/path/out/dir
ifpath=/incoming/file/path/in/dir
dfpath=/scp/destination/file/path/xyz_out
# dfpath1=/scp/destination/file/path/xyz_out #<-- if any concept can be implemented
# dfpath2=/scp/destination/file/path/abc_out #<-- if any concept can be implemented
::sleepTimes::
appt=3
fwt=5
scpt=10
The contents of app.outFile.FileWatcher.properties:
Would appreciate if any one can paste a generic schell script to archive a file with date stamp by passing the file with fullpath as parameter
For Eg. /apps/scripts/Archive_File.sh /data_home/project_home/file.txt
this should place the file in the following directory
... (8 Replies)
Hi
Please help me in this
I want to execute a shell script abc.ksh.
But I only want it to execute if file XYZ is not present.
If file XYZ is present than I want to unix to sleep for 5 Sec and than agaian check for XYX existence.
if it sleeps for more than 30 seconds ( 6 time )I want it to... (3 Replies)
Hi all,
I have a requirement to create a "superset" file out of a number of different sources with some different and some same columns.
We intend to have a manually updateable SuperSetCols.csv which would look like
"ColA","ColB","ColC","ColD","ColE","ColF","ColG"
so someday we may add... (3 Replies)
Hi,
I need a script for "FILE WATCHER"
I have in a remote FTP location server named MAIN _104
EveryDay mrng 8 i hav to search couple of folders(COL, ATT, TRA) in that FTP server (MAIN_104), and if any files are present in that folders i hav to bring back to my computer and store it anywhere... (4 Replies)
Hi Experts
I will have be having 3 types of files in directory
file1_p0_date
file1_p1_date
file1_p2_date
As soon as it sees any of the files it needs to kick off another process and also would need the file name
For this I am creating a file watcher script which will look for file1* My... (2 Replies)
Hi all,
I am writing a script to do the following:
1) Check if there are any new <csv> files (in abc directory) to process
2) If there is new file, then perform some operation on that file.
3) Move it to a different location(after step2 is successful)
4) Do further processing on the file
... (2 Replies)
Hi Guys,
I hope anyone can help me on how to create a watcher-script. The script will act as watcher wherein whenever it will find an error (say the word Error) in a file, it will prompt the user.
Please help!
br,
rymnd_12345 (3 Replies)
if test -n "$(find/data/path/ 'filename.txst' -print-quit)
then
echo "file found"
exit 0
else
echo "file not found"
exit 46
fi
So I basically looking to understand the
if test -n "$(find/data/path/ 'filename.txst' -print-quit)
line.
Pls help to elaborate what this command... (6 Replies)
All,
I am trying to create a report on the duration of an ETL load from the file arrival to the final dump in to a database for SLA's.
Does anyone have any guidance or ideas on how metadata can be extracted; information of a file: like file name, created timestamp, count of records and load... (1 Reply)