Sponsored Content
Top Forums Shell Programming and Scripting Check file availability and place flag file Post 302898355 by dhruuv369 on Monday 21st of April 2014 04:38:48 PM
Old 04-21-2014
Our ETL tool is installed on Linux and currently many of the jobs such as (a)formatting the files received by us (as per our requirement) from different sources/users, (b) placing the indicator/flag file are done manually. We are now trying to automate this process. We do not know when these files will be made available, hence we keep looking for the files received and if the files arrive, we manually place the indicator/flag file saying that the files from these and these sources are available which in turn initiates the ETL jobs depending on the flag file placed.

One source/user can send multiple source files, hence the script should look for all the source files before running its corresponding job.

See the answers for your questions below:

Where are these source file located? These are dropped in a directory on Linux by different sources/users. This directory is created specifically for the source files and serves as a location for the ETL process to pick up these files.

How is this script supposed to determine which source files it is trying to find? I was planning to use the file pattern of all these files.

What does "place flag files" mean? A specific indicator file for different sets of source files which indicates that the source files are available and the ETL job can be started to load these files into our database.

What is the name of a flag file? This will depend on the sources from which we have received the file. Based on this file ETL will start the corresponding job.

Where does the flag file go? I have created a directory where all the flag files go. ETL job will be looking for this file before kicking off the corresponding processes.

What needs to be in the flag file? An empty flag file will do. These are currently created manually by our team. We just use a touch command to create these flag files.

How long is "some time"? It is the sleep time for the script. The script should wait for this time and then start looking for the source files again. Did not decide the sleep time yet, but I think for now 5 minutes will work.

Why create a flag file instead of just kicking off the ETL job? The files that we receive are not formatted properly. This part is currently being done manually, which I am trying to take care of in an automated shell script.

How do you kick off an ETL job? ETL tool is on Linux machine, so it is using different utilities provided by the tool (eg. PMCMD) to kick off the process.

Please let me know if you need further clarification on the above questions.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

insert file 1 at a specific place of file 2

Hello, I need to search in file2 for class A : public B { and insert right after that the content of file1. I am a bit lost as to which tools (which bash functions, awk...). I should use. Thanks for some directions here. Regards (1 Reply)
Discussion started by: JCR
1 Replies

2. Shell Programming and Scripting

Shell to run one after another checking any flag or file

a.ksh & b.ksh run at the same time and takes huge resource and time.I want to put a technique so that one wil run after another. eg put a flag so that each script will check if it running , then sleep and wait it to finish. Can some one advise (3 Replies)
Discussion started by: konark
3 Replies

3. Solaris

Check Database availability?

Hi All, My script gets aborted as it is connecting as a remote database through a DB link on every 3rd sunday.As the remote DB is down for more than 6 hours. Can TNSPING DBNAME inside my script ensure the database is up??? Then how can i check the database is up or down ? ... (2 Replies)
Discussion started by: megh
2 Replies

4. Shell Programming and Scripting

using flag inside a for loop to check condition

I have a logic like this It initializes the flag variable as "T" at the beginning of the loop everytime Inside each loop it checks for two conditions and updates the flag variable as "A" or "B" In the end of the loop it checks for the value of the variable flag for "A" or "B" and execute... (4 Replies)
Discussion started by: codeman007
4 Replies

5. Programming

Some how the open(file,flag, acc) returns 0 and write to the screen, instead of the file ???

I am out of idea what to do to resolve the problem! I need to use the open(file, for.., access) function to write a file. Never have the situation like that: it is return 0 - zero. As a result all write(..) going to the screen! What the problem it could be? I do not even know... (2 Replies)
Discussion started by: alex_5161
2 Replies

6. Shell Programming and Scripting

how to check availability of data of one file in other

hi all i have two files in unix 1.table.name 2.all.tables i need to check the availability of all data of table.name in all.tables.both the table contains n number of data.i need a script which will check one by one whether the data exist in table.name or not.I want this in SH. (1 Reply)
Discussion started by: alisha
1 Replies

7. Shell Programming and Scripting

Read from file specific place in file using inode

Hello, I am using tcsh on AIX. I would like to write a script that does the following: 1. given an inode, how do I find exactly the name of the file? I know I could do this using ls -i | grep <inode> but it returns: <inode> <filename>. I need some string manipulation or something to... (1 Reply)
Discussion started by: lastZenMaster
1 Replies

8. Shell Programming and Scripting

Check File availability

Hi, I have a task to write a script , which will run everyday to check whether my system has received 2 files from source system. It should mail me ( using mailx preferably ) if the files has not arrived from the source. The scenario is everyday I get 2 files from source system.,... (3 Replies)
Discussion started by: cratercrabs
3 Replies

9. UNIX for Dummies Questions & Answers

For loop to check the files availability

Hi, I need a help with my script. Below is my script. What I am doing here is finding the date of the file which I need to process and then finding the list of file names. Based on the list of file names, check has to be done to see if all the files are there and log the details to a error... (3 Replies)
Discussion started by: Vijay81
3 Replies

10. Shell Programming and Scripting

Search for a file in all directories and place the file in that directory

Hi All, Daily I am getting the updated file. I have to search for this file in all directories and sub directories. If the file existed in a particular directory then move this updated file to that particular directory. If the file is not existed in any of the directories then place this... (4 Replies)
Discussion started by: ROCK_PLSQL
4 Replies
ExtUtils::Command(3pm)					 Perl Programmers Reference Guide				    ExtUtils::Command(3pm)

NAME
ExtUtils::Command - utilities to replace common UNIX commands in Makefiles etc. SYNOPSIS
perl -MExtUtils::Command -e cat files... > destination perl -MExtUtils::Command -e mv source... destination perl -MExtUtils::Command -e cp source... destination perl -MExtUtils::Command -e touch files... perl -MExtUtils::Command -e rm_f file... perl -MExtUtils::Command -e rm_rf directories... perl -MExtUtils::Command -e mkpath directories... perl -MExtUtils::Command -e eqtime source destination perl -MExtUtils::Command -e chmod mode files... perl -MExtUtils::Command -e test_f file DESCRIPTION
The module is used to replace common UNIX commands. In all cases the functions work from @ARGV rather than taking arguments. This makes them easier to deal with in Makefiles. perl -MExtUtils::Command -e some_command some files to work on NOT perl -MExtUtils::Command -e 'some_command qw(some files to work on)' Filenames with * and ? will be glob expanded. cat Concatenates all files mentioned on command line to STDOUT. eqtime src dst Sets modified time of dst to that of src rm_rf files.... Removes directories - recursively (even if readonly) rm_f files.... Removes files (even if readonly) touch files ... Makes files exist, with current timestamp mv source... destination Moves source to destination. Multiple sources are allowed if destination is an existing directory. cp source... destination Copies source to destination. Multiple sources are allowed if destination is an existing directory. chmod mode files... Sets UNIX like permissions 'mode' on all the files. mkpath directory... Creates directory, including any parent directories. test_f file Tests if a file exists BUGS
Should probably be Auto/Self loaded. SEE ALSO
ExtUtils::MakeMaker, ExtUtils::MM_Unix, ExtUtils::MM_Win32 AUTHOR
Nick Ing-Simmons <nick@ni-s.u-net.com>. perl v5.8.0 2002-06-01 ExtUtils::Command(3pm)
All times are GMT -4. The time now is 11:30 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy