04-21-2014
Our ETL tool is installed on Linux and currently many of the jobs such as (a)formatting the files received by us (as per our requirement) from different sources/users, (b) placing the indicator/flag file are done manually. We are now trying to automate this process. We do not know when these files will be made available, hence we keep looking for the files received and if the files arrive, we manually place the indicator/flag file saying that the files from these and these sources are available which in turn initiates the ETL jobs depending on the flag file placed.
One source/user can send multiple source files, hence the script should look for all the source files before running its corresponding job.
See the answers for your questions below:
Where are these source file located? These are dropped in a directory on Linux by different sources/users. This directory is created specifically for the source files and serves as a location for the ETL process to pick up these files.
How is this script supposed to determine which source files it is trying to find? I was planning to use the file pattern of all these files.
What does "place flag files" mean? A specific indicator file for different sets of source files which indicates that the source files are available and the ETL job can be started to load these files into our database.
What is the name of a flag file? This will depend on the sources from which we have received the file. Based on this file ETL will start the corresponding job.
Where does the flag file go? I have created a directory where all the flag files go. ETL job will be looking for this file before kicking off the corresponding processes.
What needs to be in the flag file? An empty flag file will do. These are currently created manually by our team. We just use a touch command to create these flag files.
How long is "some time"? It is the sleep time for the script. The script should wait for this time and then start looking for the source files again. Did not decide the sleep time yet, but I think for now 5 minutes will work.
Why create a flag file instead of just kicking off the ETL job? The files that we receive are not formatted properly. This part is currently being done manually, which I am trying to take care of in an automated shell script.
How do you kick off an ETL job? ETL tool is on Linux machine, so it is using different utilities provided by the tool (eg. PMCMD) to kick off the process.
Please let me know if you need further clarification on the above questions.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hello,
I need to search in file2 for
class A : public B
{
and insert right after that the content of file1.
I am a bit lost as to which tools (which bash functions, awk...). I should use. Thanks for some directions here.
Regards (1 Reply)
Discussion started by: JCR
1 Replies
2. Shell Programming and Scripting
a.ksh & b.ksh run at the same time and takes huge resource and time.I want to put a technique so that one wil run after another.
eg put a flag so that each script will check if it running , then sleep and wait it to finish.
Can some one advise (3 Replies)
Discussion started by: konark
3 Replies
3. Solaris
Hi All,
My script gets aborted as it is connecting as a remote database through a DB link on every 3rd sunday.As the remote DB is down for more than 6 hours.
Can TNSPING DBNAME inside my script ensure the database is up???
Then how can i check the database is up or down ?
... (2 Replies)
Discussion started by: megh
2 Replies
4. Shell Programming and Scripting
I have a logic like this
It initializes the flag variable as "T" at the beginning of the loop everytime
Inside each loop it checks for two conditions and updates the flag variable as "A" or "B"
In the end of the loop it checks for the value of the variable flag for "A" or "B" and execute... (4 Replies)
Discussion started by: codeman007
4 Replies
5. Programming
I am out of idea what to do to resolve the problem!
I need to use the open(file, for.., access) function to write a file.
Never have the situation like that: it is return 0 - zero.
As a result all write(..) going to the screen!
What the problem it could be?
I do not even know... (2 Replies)
Discussion started by: alex_5161
2 Replies
6. Shell Programming and Scripting
hi all
i have two files in unix
1.table.name
2.all.tables
i need to check the availability of all data of table.name in all.tables.both the table contains n number of data.i need a script which will check one by one whether the data exist in table.name or not.I want this in SH. (1 Reply)
Discussion started by: alisha
1 Replies
7. Shell Programming and Scripting
Hello, I am using tcsh on AIX.
I would like to write a script that does the following:
1. given an inode, how do I find exactly the name of the file?
I know I could do this using ls -i | grep <inode>
but it returns: <inode> <filename>. I need some string manipulation or something to... (1 Reply)
Discussion started by: lastZenMaster
1 Replies
8. Shell Programming and Scripting
Hi,
I have a task to write a script , which will run everyday to check whether my system has received 2 files from source system. It should mail me ( using mailx preferably ) if the files has not arrived from the source.
The scenario is everyday I get 2 files from source system.,... (3 Replies)
Discussion started by: cratercrabs
3 Replies
9. UNIX for Dummies Questions & Answers
Hi,
I need a help with my script.
Below is my script. What I am doing here is finding the date of the file which I need to process and then finding the list of file names.
Based on the list of file names, check has to be done to see if all the files are there and log the details to a error... (3 Replies)
Discussion started by: Vijay81
3 Replies
10. Shell Programming and Scripting
Hi All,
Daily I am getting the updated file.
I have to search for this file in all directories and sub directories.
If the file existed in a particular directory then move this updated file to that particular directory.
If the file is not existed in any of the directories then place this... (4 Replies)
Discussion started by: ROCK_PLSQL
4 Replies
LEARN ABOUT REDHAT
extutils::command
ExtUtils::Command(3pm) Perl Programmers Reference Guide ExtUtils::Command(3pm)
NAME
ExtUtils::Command - utilities to replace common UNIX commands in Makefiles etc.
SYNOPSIS
perl -MExtUtils::Command -e cat files... > destination
perl -MExtUtils::Command -e mv source... destination
perl -MExtUtils::Command -e cp source... destination
perl -MExtUtils::Command -e touch files...
perl -MExtUtils::Command -e rm_f file...
perl -MExtUtils::Command -e rm_rf directories...
perl -MExtUtils::Command -e mkpath directories...
perl -MExtUtils::Command -e eqtime source destination
perl -MExtUtils::Command -e chmod mode files...
perl -MExtUtils::Command -e test_f file
DESCRIPTION
The module is used to replace common UNIX commands. In all cases the functions work from @ARGV rather than taking arguments. This makes
them easier to deal with in Makefiles.
perl -MExtUtils::Command -e some_command some files to work on
NOT
perl -MExtUtils::Command -e 'some_command qw(some files to work on)'
Filenames with * and ? will be glob expanded.
cat Concatenates all files mentioned on command line to STDOUT.
eqtime src dst
Sets modified time of dst to that of src
rm_rf files....
Removes directories - recursively (even if readonly)
rm_f files....
Removes files (even if readonly)
touch files ...
Makes files exist, with current timestamp
mv source... destination
Moves source to destination. Multiple sources are allowed if destination is an existing directory.
cp source... destination
Copies source to destination. Multiple sources are allowed if destination is an existing directory.
chmod mode files...
Sets UNIX like permissions 'mode' on all the files.
mkpath directory...
Creates directory, including any parent directories.
test_f file
Tests if a file exists
BUGS
Should probably be Auto/Self loaded.
SEE ALSO
ExtUtils::MakeMaker, ExtUtils::MM_Unix, ExtUtils::MM_Win32
AUTHOR
Nick Ing-Simmons <nick@ni-s.u-net.com>.
perl v5.8.0 2002-06-01 ExtUtils::Command(3pm)