Sponsored Content
Top Forums Shell Programming and Scripting Picking up files conditionally Post 302953180 by Saanvi1 on Tuesday 25th of August 2015 04:07:41 PM
Old 08-25-2015
It is one of the ETL servers that loads these files into databases. Each file pertains to different load process. So passing filename in parrallel will load various tables based on filename simultaneously.
Currently it is working fine in production without any issues. But ocassionally now we started receiving multiple files( not high in number) with same name but different dates, hence kicking off same file will execute the same ETL code causing it to fail multiple times. I am trying to avoid a situation of failure and want to keep the parallel execution in place for individual files and for the ones that are more than one file with same name that needs to be sequential load one after the other.

Code:
 
 1.    app-cnd-imp-20150820.txt
 2.    app-cxyzm-imp-20150820.txt
 3.    app-petco-imp-20150820.txt
 4.    app-mobility-imp-20150820.txt
 5.    app-mobility-imp-20150821.txt
 6.    app-mobility-imp-20150822.txt
 7.    app-cellular-imp-20150824.txt


So in the file list above I can run file number 1,2,3,4,7 in one pass of loop and wait for completion and number 5 file in second pass of the loop and wait for completion and number 6 file in third pass of loop as the 4, 5,6 pertains to the same ETL code and will fail the load process.

Thanks

Last edited by Saanvi1; 08-25-2015 at 05:11 PM.. Reason: spelling correction
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Email from script conditionally

I have a script that is run from the Cron 3 times an hour, here is the cron line: 02,22,42 7-18 * * 1-5 /hci/TEST/bin/myscript.ksh TEST 1>/hci/TEST/logs/myscript.info 2>/hci/TEST/logs/myscript I am curious as to whether the time parameters from cron, ( 02, 22, 42 etc) can be accessed from the... (2 Replies)
Discussion started by: dfb500
2 Replies

2. UNIX for Dummies Questions & Answers

Conditionally joining lines in vi

I've done this before but I can't remember how. Too long away from vi. I want to do a search are replace, but I want the replace to be a join. Example see spot run see spot walk see spot run fast see spot hop %s/run$/<somehow perform a join with the next line>/g so the results... (0 Replies)
Discussion started by: ifermon
0 Replies

3. Shell Programming and Scripting

Get min from a column conditionally

hi, i have a file with folowing content: STORAGE PERCENTAGE FLAG: /storage_01 64% 0 /storage_02 17% 1 /storage_03 10% 0 /storage_04 50% 1 I need to get the value of STORAGE from those with FLAG=0 and which has the min PERCENTAGE i am able to get the STORAGE corresponding to... (8 Replies)
Discussion started by: kichu
8 Replies

4. Shell Programming and Scripting

Conditionally delete last X lines

delete last X lines,which start with + example file: test1 test2 remove1 remove2 one liner shell is preferred. (8 Replies)
Discussion started by: honglus
8 Replies

5. Shell Programming and Scripting

conditionally combine text from two files into one

Hi! I'm trying to take multiple text files (6), which have text on some lines but not others, and combine them. I'd also like to make the values in one column of some of the files (files 4-6) negative. I'm trying to write a short script (see below) as I have to do this with a large number of... (2 Replies)
Discussion started by: felix.echidna
2 Replies

6. UNIX for Dummies Questions & Answers

How to conditionally replace a pattern?

Hi, How to replace only the function calls with a new name and skip the function definition and declarations. consider the following code. There are 2 functions defined here returnint and returnvoid. I need to replace returnint with giveint and returnvoid with givevoid only in the function... (1 Reply)
Discussion started by: i.srini89
1 Replies

7. Shell Programming and Scripting

How to conditionally replace a pattern?

Hi, How to replace only the function calls with a new name and skip the function definition and declarations. consider the following code. There are 2 functions defined here returnint and returnvoid. I need to replace returnint with giveint and returnvoid with givevoid only in the function... (2 Replies)
Discussion started by: i.srini89
2 Replies

8. Shell Programming and Scripting

Urgent ...pls Sorting files based on timestamp and picking the latest file

Hi Friends, Newbie to shell scripting. Currently i have used the below to sort data based on filenames and datestamp $ printf '%s\n' *.dat* | sort -t. -k3,4 filename_1.dat.20120430.Z filename_2.dat.20120430.Z filename_3.dat.20120430.Z filename_1.dat.20120501.Z filename_2.dat.20120501.Z... (1 Reply)
Discussion started by: robertbrown624
1 Replies

9. UNIX for Dummies Questions & Answers

Sorting files based on timestamp and picking the latest file

Hi Friends, Newbie to shell scripting Currently i have used the below to sort data based on filenames and datestamp $ printf '%s\n' *.dat* | sort -t. -k3,4 filename_1.dat.20120430.Z filename_2.dat.20120430.Z filename_3.dat.20120430.Z filename_1.dat.20120501.Z filename_2.dat.20120501.Z... (12 Replies)
Discussion started by: robertbrown624
12 Replies

10. Shell Programming and Scripting

Picking up last two specific files, for transfer

Hi I need to transfer files with csv extension from one server to another (all running solaris). But the files to be picked are in a directory that also has: /directory deposit_698.csv deposit_698.checksum deposit_699.csv deposit_699.checksum deposit_700.csv deposit_700.checksum... (15 Replies)
Discussion started by: fretagi
15 Replies
DH_INSTALLDEB(1)						     Debhelper							  DH_INSTALLDEB(1)

NAME
dh_installdeb - install files into the DEBIAN directory SYNOPSIS
dh_installdeb [debhelperoptions] DESCRIPTION
dh_installdeb is a debhelper program that is responsible for installing files into the DEBIAN directories in package build directories with the correct permissions. FILES
package.postinst package.preinst package.postrm package.prerm These maintainer scripts are installed into the DEBIAN directory. Inside the scripts, the token #DEBHELPER# is replaced with shell script snippets generated by other debhelper commands. package.triggers package.shlibs These control files are installed into the DEBIAN directory. Note that package.shlibs is only installed in compat level 9 and earlier. In compat 10, please use dh_makeshlibs(1). package.conffiles This control file will be installed into the DEBIAN directory. In v3 compatibility mode and higher, all files in the etc/ directory in a package will automatically be flagged as conffiles by this program, so there is no need to list them manually here. package.maintscript Lines in this file correspond to dpkg-maintscript-helper(1) commands and parameters. However, the "maint-script-parameters" should not be included as debhelper will add those automatically. Example: # Correct rm_conffile /etc/obsolete.conf 0.2~ foo # INCORRECT rm_conffile /etc/obsolete.conf 0.2~ foo -- "$@" In compat 10 or later, any shell metacharacters will be escaped, so arbitrary shell code cannot be inserted here. For example, a line such as "mv_conffile /etc/oldconffile /etc/newconffile" will insert maintainer script snippets into all maintainer scripts sufficient to move that conffile. It was also the intention to escape shell metacharacters in previous compat levels. However, it did not work properly and as such it was possible to embed arbitrary shell code in earlier compat levels. The dh_installdeb tool will do some basic validation of some of the commands listed in this file to catch common mistakes. The validation is enabled as a warning since compat 10 and as a hard error in compat 12. SEE ALSO
debhelper(7) This program is a part of debhelper. AUTHOR
Joey Hess <joeyh@debian.org> 11.1.6ubuntu2 2018-05-10 DH_INSTALLDEB(1)
All times are GMT -4. The time now is 11:52 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy