Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Check file availability and place flag file
Top Forums Shell Programming and Scripting Check file availability and place flag file Post 302898355 by dhruuv369 on Monday 21st of April 2014 04:38:48 PM
Old 04-21-2014
Our ETL tool is installed on Linux and currently many of the jobs such as (a)formatting the files received by us (as per our requirement) from different sources/users, (b) placing the indicator/flag file are done manually. We are now trying to automate this process. We do not know when these files will be made available, hence we keep looking for the files received and if the files arrive, we manually place the indicator/flag file saying that the files from these and these sources are available which in turn initiates the ETL jobs depending on the flag file placed.

One source/user can send multiple source files, hence the script should look for all the source files before running its corresponding job.

See the answers for your questions below:

Where are these source file located? These are dropped in a directory on Linux by different sources/users. This directory is created specifically for the source files and serves as a location for the ETL process to pick up these files.

How is this script supposed to determine which source files it is trying to find? I was planning to use the file pattern of all these files.

What does "place flag files" mean? A specific indicator file for different sets of source files which indicates that the source files are available and the ETL job can be started to load these files into our database.

What is the name of a flag file? This will depend on the sources from which we have received the file. Based on this file ETL will start the corresponding job.

Where does the flag file go? I have created a directory where all the flag files go. ETL job will be looking for this file before kicking off the corresponding processes.

What needs to be in the flag file? An empty flag file will do. These are currently created manually by our team. We just use a touch command to create these flag files.

How long is "some time"? It is the sleep time for the script. The script should wait for this time and then start looking for the source files again. Did not decide the sleep time yet, but I think for now 5 minutes will work.

Why create a flag file instead of just kicking off the ETL job? The files that we receive are not formatted properly. This part is currently being done manually, which I am trying to take care of in an automated shell script.

How do you kick off an ETL job? ETL tool is on Linux machine, so it is using different utilities provided by the tool (eg. PMCMD) to kick off the process.

Please let me know if you need further clarification on the above questions.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

insert file 1 at a specific place of file 2

Hello, I need to search in file2 for class A : public B { and insert right after that the content of file1. I am a bit lost as to which tools (which bash functions, awk...). I should use. Thanks for some directions here. Regards (1 Reply)
Discussion started by: JCR
1 Replies

2. Shell Programming and Scripting

Shell to run one after another checking any flag or file

a.ksh & b.ksh run at the same time and takes huge resource and time.I want to put a technique so that one wil run after another. eg put a flag so that each script will check if it running , then sleep and wait it to finish. Can some one advise (3 Replies)
Discussion started by: konark
3 Replies

3. Solaris

Check Database availability?

Hi All, My script gets aborted as it is connecting as a remote database through a DB link on every 3rd sunday.As the remote DB is down for more than 6 hours. Can TNSPING DBNAME inside my script ensure the database is up??? Then how can i check the database is up or down ? ... (2 Replies)
Discussion started by: megh
2 Replies

4. Shell Programming and Scripting

using flag inside a for loop to check condition

I have a logic like this It initializes the flag variable as "T" at the beginning of the loop everytime Inside each loop it checks for two conditions and updates the flag variable as "A" or "B" In the end of the loop it checks for the value of the variable flag for "A" or "B" and execute... (4 Replies)
Discussion started by: codeman007
4 Replies

5. Programming

Some how the open(file,flag, acc) returns 0 and write to the screen, instead of the file ???

I am out of idea what to do to resolve the problem! I need to use the open(file, for.., access) function to write a file. Never have the situation like that: it is return 0 - zero. As a result all write(..) going to the screen! What the problem it could be? I do not even know... (2 Replies)
Discussion started by: alex_5161
2 Replies

6. Shell Programming and Scripting

how to check availability of data of one file in other

hi all i have two files in unix 1.table.name 2.all.tables i need to check the availability of all data of table.name in all.tables.both the table contains n number of data.i need a script which will check one by one whether the data exist in table.name or not.I want this in SH. (1 Reply)
Discussion started by: alisha
1 Replies

7. Shell Programming and Scripting

Read from file specific place in file using inode

Hello, I am using tcsh on AIX. I would like to write a script that does the following: 1. given an inode, how do I find exactly the name of the file? I know I could do this using ls -i | grep <inode> but it returns: <inode> <filename>. I need some string manipulation or something to... (1 Reply)
Discussion started by: lastZenMaster
1 Replies

8. Shell Programming and Scripting

Check File availability

Hi, I have a task to write a script , which will run everyday to check whether my system has received 2 files from source system. It should mail me ( using mailx preferably ) if the files has not arrived from the source. The scenario is everyday I get 2 files from source system.,... (3 Replies)
Discussion started by: cratercrabs
3 Replies

9. UNIX for Dummies Questions & Answers

For loop to check the files availability

Hi, I need a help with my script. Below is my script. What I am doing here is finding the date of the file which I need to process and then finding the list of file names. Based on the list of file names, check has to be done to see if all the files are there and log the details to a error... (3 Replies)
Discussion started by: Vijay81
3 Replies

10. Shell Programming and Scripting

Search for a file in all directories and place the file in that directory

Hi All, Daily I am getting the updated file. I have to search for this file in all directories and sub directories. If the file existed in a particular directory then move this updated file to that particular directory. If the file is not existed in any of the directories then place this... (4 Replies)
Discussion started by: ROCK_PLSQL
4 Replies

LEARN ABOUT ULTRIX

lprm

lprm(1) 						      General Commands Manual							   lprm(1)

Name
       lprm - remove jobs from line printer queue

Syntax
       lprm [-Pprinter] [-] [job #...] [user...]

Description
       The  command removes a job, or jobs, from a printer's spool queue.  Since the spooling directory is protected from users, using is normally
       the only method by which a user may remove a job.

       The command without any arguments deletes the currently active job if it is owned by the user who invoked

       If the - flag is specified, removes all jobs which a user owns.	If the super-user employs this flag, the spool queue is emptied  entirely.
       The owner is determined by the user's login name and host name on the machine where the command was invoked.

       Specifying  a user's name, or list of user names, causes to attempt to remove any jobs queued belonging to that user (or users).  This form
       of invoking is useful only to the super-user.

       A user may dequeue an individual job by specifying its job number.  This number may be obtained from the program.  For example,
       % lpq -l

       1st: ken  [job #013ucbarpa]
	    (standard input)	100 bytes
       % lprm 13

       The command announces the names of any files it removes and is silent if there are no jobs in the queue which match the request list.

       The command kills off an active daemon, if necessary, before removing any spooling files.  If a daemon is killed, a new	one  is  automati-
       cally restarted upon completion of file removals.

Options
       -		   Removes all jobs owned by you only.

       -P printer	   Removes jobs from specified printer.  It may be used to specify the queue associated with a specific printer (otherwise
			   the default printer, or the value of the PRINTER variable in the environment is used).

Restrictions
       Since there are race conditions possible in the update of the lock file, the currently active job may be incorrectly identified.

Diagnostics
       ``Permission denied" if the user tries to remove files other than his own.

Files
       /etc/printcap	   printer characteristics file
       /usr/spool/*	   spooling directories
       /usr/spool/*/lock   lock file used to obtain the pid of the current
			   daemon and the job number of the currently active job

See Also
       lpq(1), lpr(1), lpd(8)

																	   lprm(1)
All times are GMT -4. The time now is 05:13 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy