I have to check a directory on Linux (via shell Script which I am trying to build) for about 20 different source files with file patterns and if the files are made available in the directory, I should place flag files for which my other ETL jobs are waiting on to kick off. If the source files are not available in the desired directory, then the script should sleep for some time and then try looking for these files again. However, since the filenames changes every week with its week numbers or timestamps on the files eg filename04202014.CSV for this week and filename04272014.CSV for next week, I am having a bit of challenge to create this "Kicker script". Any good suggestions on how this can be achieved?
Our ETL tool is installed on Linux and currently many of the jobs such as (a)formatting the files received by us (as per our requirement) from different sources/users, (b) placing the indicator/flag file are done manually. We are now trying to automate this process. We do not know when these files will be made available, hence we keep looking for the files received and if the files arrive, we manually place the indicator/flag file saying that the files from these and these sources are available which in turn initiates the ETL jobs depending on the flag file placed.
One source/user can send multiple source files, hence the script should look for all the source files before running its corresponding job.
See the answers for your questions below:
Where are these source file located? These are dropped in a directory on Linux by different sources/users. This directory is created specifically for the source files and serves as a location for the ETL process to pick up these files.
How is this script supposed to determine which source files it is trying to find? I was planning to use the file pattern of all these files.
What does "place flag files" mean? A specific indicator file for different sets of source files which indicates that the source files are available and the ETL job can be started to load these files into our database.
What is the name of a flag file? This will depend on the sources from which we have received the file. Based on this file ETL will start the corresponding job.
Where does the flag file go? I have created a directory where all the flag files go. ETL job will be looking for this file before kicking off the corresponding processes.
What needs to be in the flag file? An empty flag file will do. These are currently created manually by our team. We just use a touch command to create these flag files.
How long is "some time"? It is the sleep time for the script. The script should wait for this time and then start looking for the source files again. Did not decide the sleep time yet, but I think for now 5 minutes will work.
Why create a flag file instead of just kicking off the ETL job? The files that we receive are not formatted properly. This part is currently being done manually, which I am trying to take care of in an automated shell script.
How do you kick off an ETL job? ETL tool is on Linux machine, so it is using different utilities provided by the tool (eg. PMCMD) to kick off the process.
Please let me know if you need further clarification on the above questions.
I wanted specifics. You provided generalities. With what you have given us, the script needs to be something like:
Code:
cd somewhere
while [ 1 ]
do for file in somepattern
do if [ ! FlagFileExistsCorrespondingTo$file ]
then CreateFlagFileCorrespondingTo$file
fi
done
if [ AllExpectedFilesFound ]
then break
else sleep 300
fi
done
This User Gave Thanks to Don Cragun For This Post:
@Don, I tried the script in the following way, but I am stuck in the middle part where I need to check the conditions for each request flag separately. Could you please help me out.
Code:
while [ 2 -ge 1 ]
do
if [ -f /dir1/dir2/dir3/break.timx ]
then
echo "The Script is ending as a request has been made to break the script"
exit $SUCCESS
else
req_file_list=`ls /dir1/dir2/dir3/ReqFiles`
echo $req_file_list
atleast_one_req_file_processed=0
for req_file in $req_file_list
do
if [ $atleast_one_req_file_processed -eq 0 ]
then
grep $req_file /dir1/dir2/dir3/flags_info.txt > /dir1/dir2/dir3/req_file_current_session.txt
# /dir1/dir2/dir3/flags_info.txt is a fixed file and has all the information on the request flags and the conditions the script expects and should look on before placing the indicator/flag file.
atleast_one_req_file_processed=`expr $atleast_one_req_file_processed + 1`
echo $atleast_one_req_file_processed
else
grep $req_file /dir1/dir2/dir3/flags_info.txt >> /dir1/dir2/dir3/req_file_current_session.txt
fi
done
Stuck at this part, as I am not sure how to check conditions for each request file separately
sleep 300
fi
done
Example of the /dir1/dir2/dir3/flags_info.txt is given below:
I'm afraid I cannot help you as the logics required escape me. I don't see the directories' structures, nor the request files / flags names, nor what they trigger, and when. The flags_info.txt file is fixed, I presume. Do you need/use its contents to schedule the ETL jobs?
Anyhow, there quite some opportunities in your script above:
- change your while and if to until [ -f ...break.timx]
- you don't need the req_file_list variable, nor the for loop. Depending on your shell, you might grep for the ls result immediately, and set the at_least_one variable according to its exit code.
Try
Code:
until [ -f /dir1/dir2/dir3/break.timx ]
do ls /dir1/dir2/dir3/ReqFiles |grep -f - /dir1/dir2/dir3/flags_info.txt >xx
atleast_one_req_file_processed=$?
done
If you describe your needs a bit more in detail, we might be able to add value to your problem.
Hi All,
Daily I am getting the updated file.
I have to search for this file in all directories and sub directories.
If the file existed in a particular directory then move this updated file to that particular directory.
If the file is not existed in any of the directories then place this... (4 Replies)
Hi,
I need a help with my script.
Below is my script. What I am doing here is finding the date of the file which I need to process and then finding the list of file names.
Based on the list of file names, check has to be done to see if all the files are there and log the details to a error... (3 Replies)
Hi,
I have a task to write a script , which will run everyday to check whether my system has received 2 files from source system. It should mail me ( using mailx preferably ) if the files has not arrived from the source.
The scenario is everyday I get 2 files from source system.,... (3 Replies)
Hello, I am using tcsh on AIX.
I would like to write a script that does the following:
1. given an inode, how do I find exactly the name of the file?
I know I could do this using ls -i | grep <inode>
but it returns: <inode> <filename>. I need some string manipulation or something to... (1 Reply)
hi all
i have two files in unix
1.table.name
2.all.tables
i need to check the availability of all data of table.name in all.tables.both the table contains n number of data.i need a script which will check one by one whether the data exist in table.name or not.I want this in SH. (1 Reply)
I am out of idea what to do to resolve the problem!
I need to use the open(file, for.., access) function to write a file.
Never have the situation like that: it is return 0 - zero.
As a result all write(..) going to the screen!
What the problem it could be?
I do not even know... (2 Replies)
I have a logic like this
It initializes the flag variable as "T" at the beginning of the loop everytime
Inside each loop it checks for two conditions and updates the flag variable as "A" or "B"
In the end of the loop it checks for the value of the variable flag for "A" or "B" and execute... (4 Replies)
Hi All,
My script gets aborted as it is connecting as a remote database through a DB link on every 3rd sunday.As the remote DB is down for more than 6 hours.
Can TNSPING DBNAME inside my script ensure the database is up???
Then how can i check the database is up or down ?
... (2 Replies)
a.ksh & b.ksh run at the same time and takes huge resource and time.I want to put a technique so that one wil run after another.
eg put a flag so that each script will check if it running , then sleep and wait it to finish.
Can some one advise (3 Replies)
Hello,
I need to search in file2 for
class A : public B
{
and insert right after that the content of file1.
I am a bit lost as to which tools (which bash functions, awk...). I should use. Thanks for some directions here.
Regards (1 Reply)
04-21-2014
