|
|
Sponsored Content
Special Forums
UNIX and Linux Applications
High Performance Computing
Parallel Execution on Multiple System
Post 302285241 by 123an on Sunday 8th of February 2009 01:44:06 AM
02-08-2009
More information regarding Parallel Execution
Hi quirkasaurus,
Thanks for ur reply..
As per ur reply i would like to give some more details about my problem,
I have 500 different set of arguments in a file(say list.txt). These arguments i need to pass to an executable (or an application; say "./applictn.out") which wil run and does the job taking each arguments. That means i will get 500 sets of outputs on executing whole system.
If I execute this using script serially(means one after other) execution wil take more time. so within a system i can execute this in parallel (i.e making them as background process) using xargs. Like
Syntax:xargs <utility> <arguments>
Given: xargs ./applictn.out list.txt..
This wil be executed in parallel on single system..
Now my problem is if i want to execute these sets(set of 500 arguments) on different linux systems, which should run in parallel. So that i should get output at lower amount of time.
parallelism i mean to say is, if mahine1 should take say 150 sets and start processing..
Machine 2 should take say 200 sets and starts processing..
machine 3 should take remaining in sets and should start working on it..
All machines should work in parallel...
Thanks..
123an
Thanks for ur reply..
As per ur reply i would like to give some more details about my problem,
I have 500 different set of arguments in a file(say list.txt). These arguments i need to pass to an executable (or an application; say "./applictn.out") which wil run and does the job taking each arguments. That means i will get 500 sets of outputs on executing whole system.
If I execute this using script serially(means one after other) execution wil take more time. so within a system i can execute this in parallel (i.e making them as background process) using xargs. Like
Syntax:xargs <utility> <arguments>
Given: xargs ./applictn.out list.txt..
This wil be executed in parallel on single system..
Now my problem is if i want to execute these sets(set of 500 arguments) on different linux systems, which should run in parallel. So that i should get output at lower amount of time.
parallelism i mean to say is, if mahine1 should take say 150 sets and start processing..
Machine 2 should take say 200 sets and starts processing..
machine 3 should take remaining in sets and should start working on it..
All machines should work in parallel...
Thanks..
123an
Quote:
not enough info for me...
are you talking about the same script with 500 different sets of arguments?
do you have the arguments already somewhere or
will you generate them?
should the jobs run in series once on their individual machine?
or can they run concurrently with a maximum threshold?
is this just for benchmarking? or is this going to be a permanent run and
everything should take about the same time?
i'm thinking . . . . just create all the command lines....
dump them all into a file...
then have another script, read this file,
divide them equally into scripts for each machine....
rcp these scripts to the respective machines,
and kick 'em off.
are you talking about the same script with 500 different sets of arguments?
do you have the arguments already somewhere or
will you generate them?
should the jobs run in series once on their individual machine?
or can they run concurrently with a maximum threshold?
is this just for benchmarking? or is this going to be a permanent run and
everything should take about the same time?
i'm thinking . . . . just create all the command lines....
dump them all into a file...
then have another script, read this file,
divide them equally into scripts for each machine....
rcp these scripts to the respective machines,
and kick 'em off.
|
|
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hello,
I wish to run parallel process forked from one script.
Currently I doing is submit them in background.
For example:
---------------------------------------------------------------
#!/usr/bin/ksh
process1 &
process2 &
process3 &
.....
.....
#here I check for completion of... (4 Replies)
Discussion started by: RishiPahuja
4 Replies
2. AIX
Hi,
Is there any way to run parallel jobs using make command?
I am using non-GNU make utility on AIX 4.3.
I want to run 2 jobs symultaneously using make utility.
Thanks.
Suman (0 Replies)
Discussion started by: suman_jakkula
0 Replies
3. Shell Programming and Scripting
Hi All,
At present I am using a UNIX Script which is running a set of JOBS. These JOBS are to be repeated for 20 times, means the same set of JOBS are repeated by just passing different arguments (From 1 to 20).
Is there any way by which I can execute them parallel?
At present its all... (4 Replies)
Discussion started by: Prashantckc
4 Replies
4. Shell Programming and Scripting
how can i process jobs parallel with conditions below.
Script1.ksh
Script2.ksh
Script3.ksh
Script4.ksh
Script5.ksh
Script6.ksh
Script7.ksh
Script8.ksh
Script9.ksh
Script10.ksh
After successful completion of Script1.ksh I need to run Script7.ksh.
After successful... (4 Replies)
Discussion started by: ford2020
4 Replies
5. Shell Programming and Scripting
I am haveing 2 scripts, 1st script calls 2ed script for each parameter.(parameters are kept in a different txt file)
1st script
for x in `cat Export_Tables_List.txt`
do
sh Exp_Table.sh $x &
done
echo -e "1) following tables are successfully exported : \n" > temp
cat... (1 Reply)
Discussion started by: sbmk_design
1 Replies
6. Programming
Since there've been a few requests for a method to execute commands on multiple CPUs (logical or physical), with various levels of shell-, make-, or Perl-based solutions, ranging from well-done to well-meant, and mostly specific to a certain problem, I've started to write a C-based solution... (4 Replies)
Discussion started by: pludi
4 Replies
7. UNIX for Advanced & Expert Users
Hi All,
We have a table that has to store around 80-100 million records. The table is partitioned by a column called Market Code. There are 30 partitions each corresponding to one MRKT_CD.
The source of this table is a join between 3-4 other tables. We are loading this table through SQLPLUS... (2 Replies)
Discussion started by: jerome_rajan
2 Replies
8. Shell Programming and Scripting
I have few very huge files ~ 2 Billion rows of 130 column(CDR data) in a folder, I have written shell script need to read on each file in a folder and will create a new files based on some logic.
But problem is it's taking time to create a new file due to the size , So i dont want to corrupt... (6 Replies)
Discussion started by: rspwilliam
6 Replies
9. Homework & Coursework Questions
I have a requirement (in a shell script) to connect to several servers at once and execute a series of commands.
I am aware that ssh can be used for sequential execution. But since most of the commands that I need to execute take a long time, I have to go for the parallel option.
Is there... (2 Replies)
Discussion started by: sneha1887
2 Replies
10. UNIX for Beginners Questions & Answers
i have say x number of procedure to run, ie i have one procedure which accepts variable and i need that to run in parallel and capture the error code if in case if it fails through the unix.
sqlplus <EOF> exec test_t (abc,124); </EOF>
sqlplus <EOF> exec test_t (abc,125); </EOF>
sqlplus <EOF>... (2 Replies)
Discussion started by: ATWC
2 Replies
LEARN ABOUT SUSE
startpar
STARTPAR(8) System Manager's Manual STARTPAR(8) NAME
startpar - start runlevel scripts in parallel SYNOPSIS
startpar [-p par] [-i iorate] [-t timeout] [-T global_timeout] [-a arg] prg1 prg2 ... startpar [-p par] [-i iorate] [-t timeout] [-T global_timeout] -M [ boot|start|stop] DESCRIPTION
startpar is used to run multiple run-level scripts in parallel. The degree of parallelism on one CPU can be set with the -p option, the default is full parallelism. An argument to all of the scripts can be provided with the -a option. Processes block by pending I/O will weighting by the factor 800. To change this factor the option -i can be used to specify an other value. The output of each script is buffered and written when the script exits, so output lines of different scripts won't mix. You can modify this behaviour by setting a timeout. The timeout set with the -t option is used as buffer timeout. If the output buffer of a script is not empty and the last output was timeout seconds ago, startpar will flush the buffer. The -T option timeout works more globally. If no output is printed for more than global_timeout seconds, startpar will flush the buffer of the script with the oldest output. Afterwards it will only print output of this script until it is finished. The -M option switches startpar into a make(1) like behaviour. This option takes three different arguments: boot, start, and stop for reading .depend.boot or .depend.start or .depend.stop respectively in the directory /etc/init.d/. By scanning the boot and runlevel direc- tories in /etc/init.d/ it then executes the appropriate scripts in parallel. FILES
/etc/init.d/.depend.boot /etc/init.d/.depend.start /etc/init.d/.depend.stop SEE ALSO
init.d(7), insserv(8), startproc(8). COPYRIGHT
2003,2004 SuSE Linux AG, Nuernberg, Germany. 2007 SuSE LINUX Products GmbH, Nuernberg, Germany. AUTHOR
Michael Schroeder <mls@suse.de> Takashi Iwai <tiwai@suse.de> Werner Fink <werner@suse.de> Jun 2003 STARTPAR(8)