Sponsored Content
Top Forums Shell Programming and Scripting Running batches of files at a time from a script Post 302419100 by ladyAnne on Thursday 6th of May 2010 09:47:15 AM
Old 05-06-2010
Running batches of files at a time from a script

Hi

I have a script that performs a process on a file.

I want to know how to include a function to run a batch of files?

Here is my script

Code:
 
#!/bin/bash
#---------------------------------------------------------------------------------------------------------------------- 
#This script allows the user to import a dsx file 
#
# Usage:
#./import.sh -s -d -r -o -f
#
#-----------------------------------------------------------------------------------------------------------------------
dbname="$dbname"
dsn="$dsn"
dsxfile="$dsxfile"
oprange=""
norule=""
verbose=""
reverb=""
module=`basename $0`
DSXLOG_MODULE=$module; export DSXLOG_MODULE
dsxdir="/var/local/dsx/import"
dsximp="/usr/local/bin/dsximp.exe"
dsxpids="/var/run/"
importdir="/var/local/bin/Automated_Testing/Auto_Scripts"

I wrote something like this:
Code:
function batch
{
for dsxfile in $dsxdir/*
do
$dsximp $norule $oprange --dsn=$dsn --dbname=$dbname --user=datasafe --password=datasafe --dsxfile=$dsxdir/$dsxfile
done

Code:
}
function displayHelp()
{
echo ""
echo " Usage: ./import.sh [ -s -d -r -o -f ]"
echo ""
echo " -s, --dsn identified by the Data Source Name "
echo " -d, --dbname Identified by the Database Name "
echo " -r, --norule Do not excecute dsxrule " 
echo " -o, --oprange Use optimize range checking " 
echo " -f, --dsxfile Identified by the file in dsx format "
echo " -b, --batch Runs the import for a batch of files "
echo " -h, --help Identified by the help menu "
echo ""

}
if [ -e ${dsxpids}$module.pid ] 
then
dsxlog $reverb --warning --module="$module" " pid = $$ : Is already running"
true
exit;
else
dsxlog $reverb --info --module="$module" "started pid = $$" 
while getopts " s: d: r o f: b e h " option
do
case $option in

f ) dsxfile="$OPTARG";;
d ) dbname="$OPTARG";;
s ) dsn="$OPTARG";;
r ) norule="-r";;
o ) oprange="-O";;
e ) verbose="--verbose";
reverb="--echo";;
b ) batch;;
h ) displayHelp;
dsxlog $reverb --info --module="$module" "displaying --helpmenu";
dsxlog $reverb --info --module="$module" "ended, pid = $$";
exit;;
? | * )displayHelp;
dsxlog $reverb --info --module="$module" "invalid option entered --see helpmenu ";
dsxlog $reverb --info --module="$module" "ended, pid = $$";
exit;;

esac;
done
if [ $# -eq 0 ]; then
displayHelp
dsxlog $reverb --info --module="$module" "no options have been entered --see helpmenu"
dsxlog $reverb --info --module="$module" "ended, pid = $$" 
exit 1
fi
 
$dsximp $norule $oprange --dsn=$dsn --dbname=$dbname --user=datasafe --password=datasafe --dsxfile=$dsxdir/$dsxfile
dsxlog $reverb --info --module="$module" "ended, pid = $$" 
fi

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need to know abt script that invokes batches and get d log files if batches failed

hi , I need to know commands to be used in the script to invoke batches in order from other scripts and then run those batches,and how to take those logs of those batches which fails........If anyone give me a better idea to complete this entire task in a single script... (5 Replies)
Discussion started by: gopimeklord
5 Replies

2. Shell Programming and Scripting

display time required to complete running script

hi is there any way i can display a countdown time needed to run a script? like load a counter at the beginning of the script with the estimated time and display the counter decrementing till it finishes running the script? (3 Replies)
Discussion started by: npatwardhan
3 Replies

3. UNIX and Linux Applications

Processing batches of files at a time

Hi I would like to run my script to process as many as 50 files at a time. Currently my script is being called like so: ./import.sh -f filename so I want to call my script in this way and it must execute every file in the directory /var/local/dsx/import (1 Reply)
Discussion started by: ladyAnne
1 Replies

4. Emergency UNIX and Linux Support

Help to optimize script running time

Dear Forum experts I have the below script which I made to run under bash shell, it runs perfectly for low records number, let us say like 100000. when I put all records (3,000,000), it's takes hours can you please suggest anything to optimize or to run in different way :-| {OFS="|";... (6 Replies)
Discussion started by: yahyaaa
6 Replies

5. Shell Programming and Scripting

Help in running a script after a particular time

Unix Gurus, I have a requirement where the shell script needs to do specific tasks after certain period of time. Daily we receive few files in a particular folder. The script does the file renaming, pass parameters to run some web services and pushes to remote FTP location. But my... (3 Replies)
Discussion started by: shankar1dada
3 Replies

6. Shell Programming and Scripting

How to know the exact running time of script!

Hi All, newbie here, I'm just wondering how can i know the exact running time of my script? Please advise, THanks, (1 Reply)
Discussion started by: nikki1200
1 Replies

7. Shell Programming and Scripting

Setting time for running of the script

Dear all, I wonder if it is possible that we can run the script from time to time..I meant, it should repeat the sourcing of the script by itself? In my case, I need to source this script manually from time to time, like once in every 10 minutes. emily, (2 Replies)
Discussion started by: emily
2 Replies

8. Shell Programming and Scripting

Example of running script with time limits?

Hi, I want to write a script that does some sort of health check on the database. It will query the database for information, some query takes long and some are quick. For example, inside the script I will do something as below: #!/bin/ksh run_query_01 & run_query_02 &... (1 Reply)
Discussion started by: newbie_01
1 Replies

9. Shell Programming and Scripting

Killing the process if running for long time in script

I am running a script which will read the data from fail line by line and call the Java program by providing the arguments from the each line. The Java code is working fast for few records and for some records its getting hanged not providing response for morethan one hour. Currently am... (4 Replies)
Discussion started by: dineshaila
4 Replies

10. Shell Programming and Scripting

Bash script to take cPanel backup in batches

I'm trying to resolve the below scenario by writing bash script. On my managed linux server I have about 30 accounts and there is not enough space to generate full 30 accounts backup in one go and transfer it via SFTP to my Backup Synology Server. What I'm thinking of doing is breaking the... (3 Replies)
Discussion started by: humble_learner
3 Replies
BATCHER(8)						    InterNetNews Documentation							BATCHER(8)

NAME
batcher - Article batching for InterNetNews SYNOPSIS
batcher [-rv] [-a articles] [-A total-articles] [-b size] [-B total-size] [-i string] [-N batches] [-p process] [-s separator] host [input] DESCRIPTION
batcher reads a list of files and prepares news batches for the specified host. It is generally used to prepare UUCP feeds, but the resulting batches can be used by any application that uses rnews(8) to inject the articles. It is normally invoked by a script run out of cron that uses shlock to lock the host, followed by ctlinnd to flush the batch file. See send-uucp(8) for a front-end for batcher. batcher reads the file input, or standard input if no file is given. If input is a relative file name, it is assumed to be in pathoutgoing as set in inn.conf. Blank lines and lines starting with a number sign ("#") are ignored. All other lines in the input should consist of one or two fields separated by a single space. The first field is the storage API token of an article. The second field, if present, specifies the size of the article in bytes. By default, batches are written to standard output (which isn't very useful if more than one batch is output), but see the -p option. OPTIONS
-a articles This flag limits the number of articles included in each batch. The default is no limit. A new batch will be started when either the total bytes or the number of articles written exceeds the specified limits. -A total-articles Limits the total number of articles written for all batches. As soon as the total number of articles written to batches reaches or exceeds total-articles, all additional articles in the input will be deferred. The default is no limit. -b size This flag sets the size limit for each batch; as soon as at least this much data has been written out, a new batch will be started. The default size is 60 KB. Using "-b 0" will allow unlimited batch sizes. -B total-size Limits the total number of bytes written for all batches. As soon as the total bytes written to batches reaches or exceeds total-size, all additional articles in the input will be deferred. The default is no limit. -i string A batch starts with an identifying line to specify the unpacking method to be used on the receiving end. When this flag is used, string, followed by a newline, will be output at the start of each batch. The default is to have no initial string (under the assumption that either the processor specified with the -p flag or some other later process will add the appropriate line). -N batches Limits the total number of batches written. As soon as the number of batches written reaches or exceeds batches, all additional articles in the input will be deferred. The default is no limit. -p process By default, batches are written to standard output, which is not useful when more than one output batch is created. If this option is given, each batch will instead be fed via a pipe to the shell command process. The process argument must be an sprintf(3) format string, which may have a single %s parameter that will be replaced with the host name. A common value is: ( echo '#! gunbatch' ; exec gzip -c ) | uux - -r -z %s!rnews which generates gzip-compressed batches and feeds them to uux. -r By default, batcher reports errors to pathlog/errlog. To suppress this redirection and report errors to standard error, use the -r flag. -s separator Each article in a batch starts with a separator line that indicates the size of the article. separator must be an sprintf(3) string, which may have a single %ld in the string that will be replaced with the size of the article. If the separator is not empty, a newline will also be appended to it when it is added to the beginning of each article. The default separator is: #! rnews %ld and this should rarely be changed. -v Upon exit, batcher reports statistics via syslog. With this flag, the statistics will also be printed to standard output. EXIT STATUS
If the input is exhausted and all batches are created successfully, batcher will exit with a zero status. If any of the limits specified with -A, -B, or -N flags are reached, or if there is an error in writing a batch, batcher will try to spool the remaining input by copying it to a file as follows: o If there was no input filename, the remaining input will be copied to pathoutgoing/host. o If an input filename was given, the remaining input will be copied to a temporary file named by appending ".bch" to the end of input (and qualified by adding pathoutgoing if input was not a fully qualified path). If this happens successfully, batcher will then try to rename this temporary file to input (thus replacing input with a copy of itself with all of lines for the successfully batched articles removed). Upon receipt of an interrupt or termination signal, batcher will finish batching the current article, close the batch, and then rewrite the batch file as described above. HISTORY
Written by Rich $alz <rsalz@uunet.uu.net> for InterNetNews. Rewritten by Russ Allbery <rra@stanford.edu> in POD. $Id: batcher.pod 8774 2009-11-15 09:23:27Z iulius $ SEE ALSO
ctlinnd(8), inn.conf(5), newsfeeds(5), rnews(8), send-uucp(8), shlock(1). INN 2.5.2 2009-11-15 BATCHER(8)
All times are GMT -4. The time now is 02:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy