02-06-2009
Parallel Execution on Multiple System
Hi All,
I am working on a project where I need to execute set of arguments (around 500) on a Simulator. If I execute this on one linux (RedHat 8.0) machine it will approximately takes 2~3 days. Since I am having more linux machines am thinking of executing these on different machines in parallel(i.e. Paralle Execution).
I tried for sanity basis following option which is already posted in this forum;
i.e. Script1.sh & Script2.sh & Script3.sh . But I didn't find much savings in this option.
I also tried downloading a tool-JOBQUEUE from site "
Download jobqueue 0.04 for Linux - jobqueue is a program for executing jobs in parallel to complete all jobs as fast as possible. - Softpedia".
This is able to split execution on multiple system, but it won't take all arguments list set (like if arguments set list has 10, it will take 8/9 skips 1 r 2).
Could you please help me in solving the problem.
Thanks in advance
Regards,
123an
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hello,
I wish to run parallel process forked from one script.
Currently I doing is submit them in background.
For example:
---------------------------------------------------------------
#!/usr/bin/ksh
process1 &
process2 &
process3 &
.....
.....
#here I check for completion of... (4 Replies)
Discussion started by: RishiPahuja
4 Replies
2. AIX
Hi,
Is there any way to run parallel jobs using make command?
I am using non-GNU make utility on AIX 4.3.
I want to run 2 jobs symultaneously using make utility.
Thanks.
Suman (0 Replies)
Discussion started by: suman_jakkula
0 Replies
3. Shell Programming and Scripting
Hi All,
At present I am using a UNIX Script which is running a set of JOBS. These JOBS are to be repeated for 20 times, means the same set of JOBS are repeated by just passing different arguments (From 1 to 20).
Is there any way by which I can execute them parallel?
At present its all... (4 Replies)
Discussion started by: Prashantckc
4 Replies
4. Shell Programming and Scripting
how can i process jobs parallel with conditions below.
Script1.ksh
Script2.ksh
Script3.ksh
Script4.ksh
Script5.ksh
Script6.ksh
Script7.ksh
Script8.ksh
Script9.ksh
Script10.ksh
After successful completion of Script1.ksh I need to run Script7.ksh.
After successful... (4 Replies)
Discussion started by: ford2020
4 Replies
5. Shell Programming and Scripting
I am haveing 2 scripts, 1st script calls 2ed script for each parameter.(parameters are kept in a different txt file)
1st script
for x in `cat Export_Tables_List.txt`
do
sh Exp_Table.sh $x &
done
echo -e "1) following tables are successfully exported : \n" > temp
cat... (1 Reply)
Discussion started by: sbmk_design
1 Replies
6. Programming
Since there've been a few requests for a method to execute commands on multiple CPUs (logical or physical), with various levels of shell-, make-, or Perl-based solutions, ranging from well-done to well-meant, and mostly specific to a certain problem, I've started to write a C-based solution... (4 Replies)
Discussion started by: pludi
4 Replies
7. UNIX for Advanced & Expert Users
Hi All,
We have a table that has to store around 80-100 million records. The table is partitioned by a column called Market Code. There are 30 partitions each corresponding to one MRKT_CD.
The source of this table is a join between 3-4 other tables. We are loading this table through SQLPLUS... (2 Replies)
Discussion started by: jerome_rajan
2 Replies
8. Shell Programming and Scripting
I have few very huge files ~ 2 Billion rows of 130 column(CDR data) in a folder, I have written shell script need to read on each file in a folder and will create a new files based on some logic.
But problem is it's taking time to create a new file due to the size , So i dont want to corrupt... (6 Replies)
Discussion started by: rspwilliam
6 Replies
9. Homework & Coursework Questions
I have a requirement (in a shell script) to connect to several servers at once and execute a series of commands.
I am aware that ssh can be used for sequential execution. But since most of the commands that I need to execute take a long time, I have to go for the parallel option.
Is there... (2 Replies)
Discussion started by: sneha1887
2 Replies
10. UNIX for Beginners Questions & Answers
i have say x number of procedure to run, ie i have one procedure which accepts variable and i need that to run in parallel and capture the error code if in case if it fails through the unix.
sqlplus <EOF> exec test_t (abc,124); </EOF>
sqlplus <EOF> exec test_t (abc,125); </EOF>
sqlplus <EOF>... (2 Replies)
Discussion started by: ATWC
2 Replies
LEARN ABOUT REDHAT
anacron
ANACRON(8) Anacron Users' Manual ANACRON(8)
NAME
anacron - runs commands periodically
SYNOPSIS
anacron [-s] [-f] [-n] [-d] [-q] [-t anacrontab] [job] ...
anacron -u [-t anacrontab] [job] ...
anacron [-V|-h]
DESCRIPTION
Anacron can be used to execute commands periodically, with a frequency specified in days. Unlike cron(8), it does not assume that the
machine is running continuously. Hence, it can be used on machines that aren't running 24 hours a day, to control daily, weekly, and
monthly jobs that are usually controlled by cron.
When executed, Anacron reads a list of jobs from a configuration file, normally /etc/anacrontab (see anacrontab(5)). This file contains
the list of jobs that Anacron controls. Each job entry specifies a period in days, a delay in minutes, a unique job identifier, and a
shell command.
For each job, Anacron checks whether this job has been executed in the last n days, where n is the period specified for that job. If not,
Anacron runs the job's shell command, after waiting for the number of minutes specified as the delay parameter.
After the command exits, Anacron records the date in a special timestamp file for that job, so it can know when to execute it again. Only
the date is used for the time calculations. The hour is not used.
When there are no more jobs to be run, Anacron exits.
Anacron only considers jobs whose identifier, as specified in the anacrontab matches any of the job command-line arguments. The job argu-
ments can be shell wildcard patterns (be sure to protect them from your shell with adequate quoting). Specifying no job arguments, is
equivalent to specifying "*" (That is, all jobs will be considered).
Unless the -d option is given (see below), Anacron forks to the background when it starts, and the parent process exits immediately.
Unless the -s or -n options are given, Anacron starts jobs immediately when their delay is over. The execution of different jobs is com-
pletely independent.
If a job generates any output on its standard output or standard error, the output is mailed to the user running Anacron (usually root).
Informative messages about what Anacron is doing are sent to syslogd(8) under facility cron, priority notice. Error messages are sent at
priority error.
"Active" jobs (i.e. jobs that Anacron already decided to run and now wait for their delay to pass, and jobs that are currently being exe-
cuted by Anacron), are "locked", so that other copies of Anacron won't run them at the same time.
OPTIONS
-f Force execution of the jobs, ignoring the timestamps.
-u Only update the timestamps of the jobs, to the current date, but don't run anything.
-s Serialize execution of jobs. Anacron will not start a new job before the previous one finished.
-n Run jobs now. Ignore the delay specifications in the /etc/anacrontab file. This options implies -s.
-d Don't fork to the background. In this mode, Anacron will output informational messages to standard error, as well as to syslog.
The output of jobs is mailed as usual.
-q Suppress messages to standard error. Only applicable with -d.
-t anacrontab
Use specified anacrontab, rather than the default
-V Print version information, and exit.
-h Print short usage message, and exit.
SIGNALS
After receiving a SIGUSR1 signal, Anacron waits for running jobs, if any, to finish and then exits. This can be used to stop Anacron
cleanly.
NOTES
Make sure that the time-zone is set correctly before Anacron is started. (The time-zone affects the date). This is usually accomplished
by setting the TZ environment variable, or by installing a /usr/lib/zoneinfo/localtime file. See tzset(3) for more information.
FILES
/etc/anacrontab
Contains specifications of jobs. See anacrontab(5) for a complete description.
/var/spool/anacron
This directory is used by Anacron for storing timestamp files.
SEE ALSO
anacrontab(5), cron(8), tzset(3)
The Anacron README file.
BUGS
Anacron never removes timestamp files. Remove unused files manually.
Anacron uses up to two file descriptors for each active job. It may run out of descriptors if there are more than about 125 active jobs
(on normal kernels).
Mail comments, suggestions and bug reports to Sean 'Shaleh' Perry <shaleh@(debian.org|valinux.com)>.
AUTHOR
Anacron was originally conceived and implemented by Christian Schwarz <schwarz@monet.m.isar.de>.
The current implementation is a complete rewrite by Itai Tzur <itzur@actcom.co.il>.
The code base is currently maintained by Sean 'Shaleh' Perry <shaleh@(debian.org|valinux.com)>.
Sean 'Shaleh' Perry 2000-06-22 ANACRON(8)