Sponsored Content
Special Forums UNIX and Linux Applications High Performance Computing Parallel Execution on Multiple System Post 302284672 by 123an on Friday 6th of February 2009 04:53:55 AM
Old 02-06-2009
Parallel Execution on Multiple System

Hi All,

I am working on a project where I need to execute set of arguments (around 500) on a Simulator. If I execute this on one linux (RedHat 8.0) machine it will approximately takes 2~3 days. Since I am having more linux machines am thinking of executing these on different machines in parallel(i.e. Paralle Execution).

I tried for sanity basis following option which is already posted in this forum;
i.e. Script1.sh & Script2.sh & Script3.sh . But I didn't find much savings in this option.

I also tried downloading a tool-JOBQUEUE from site "Download jobqueue 0.04 for Linux - jobqueue is a program for executing jobs in parallel to complete all jobs as fast as possible. - Softpedia".
This is able to split execution on multiple system, but it won't take all arguments list set (like if arguments set list has 10, it will take 8/9 skips 1 r 2).

Could you please help me in solving the problem.

Thanks in advance

Regards,
123an
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parallel Execution

Hello, I wish to run parallel process forked from one script. Currently I doing is submit them in background. For example: --------------------------------------------------------------- #!/usr/bin/ksh process1 & process2 & process3 & ..... ..... #here I check for completion of... (4 Replies)
Discussion started by: RishiPahuja
4 Replies

2. AIX

Make : parallel execution

Hi, Is there any way to run parallel jobs using make command? I am using non-GNU make utility on AIX 4.3. I want to run 2 jobs symultaneously using make utility. Thanks. Suman (0 Replies)
Discussion started by: suman_jakkula
0 Replies

3. Shell Programming and Scripting

Parallel Job Execution

Hi All, At present I am using a UNIX Script which is running a set of JOBS. These JOBS are to be repeated for 20 times, means the same set of JOBS are repeated by just passing different arguments (From 1 to 20). Is there any way by which I can execute them parallel? At present its all... (4 Replies)
Discussion started by: Prashantckc
4 Replies

4. Shell Programming and Scripting

Conditional execution and parallel jobs

how can i process jobs parallel with conditions below. Script1.ksh Script2.ksh Script3.ksh Script4.ksh Script5.ksh Script6.ksh Script7.ksh Script8.ksh Script9.ksh Script10.ksh After successful completion of Script1.ksh I need to run Script7.ksh. After successful... (4 Replies)
Discussion started by: ford2020
4 Replies

5. Shell Programming and Scripting

Parallel execution of script not syncronize

I am haveing 2 scripts, 1st script calls 2ed script for each parameter.(parameters are kept in a different txt file) 1st script for x in `cat Export_Tables_List.txt` do sh Exp_Table.sh $x & done echo -e "1) following tables are successfully exported : \n" > temp cat... (1 Reply)
Discussion started by: sbmk_design
1 Replies

6. Programming

Parallel Execution of Programs

Since there've been a few requests for a method to execute commands on multiple CPUs (logical or physical), with various levels of shell-, make-, or Perl-based solutions, ranging from well-done to well-meant, and mostly specific to a certain problem, I've started to write a C-based solution... (4 Replies)
Discussion started by: pludi
4 Replies

7. UNIX for Advanced & Expert Users

Parallel Execution of Command

Hi All, We have a table that has to store around 80-100 million records. The table is partitioned by a column called Market Code. There are 30 partitions each corresponding to one MRKT_CD. The source of this table is a join between 3-4 other tables. We are loading this table through SQLPLUS... (2 Replies)
Discussion started by: jerome_rajan
2 Replies

8. Shell Programming and Scripting

How to make parallel execution on folder ?

I have few very huge files ~ 2 Billion rows of 130 column(CDR data) in a folder, I have written shell script need to read on each file in a folder and will create a new files based on some logic. But problem is it's taking time to create a new file due to the size , So i dont want to corrupt... (6 Replies)
Discussion started by: rspwilliam
6 Replies

9. Homework & Coursework Questions

Parallel execution on multiple servers in UNIX

I have a requirement (in a shell script) to connect to several servers at once and execute a series of commands. I am aware that ssh can be used for sequential execution. But since most of the commands that I need to execute take a long time, I have to go for the parallel option. Is there... (2 Replies)
Discussion started by: sneha1887
2 Replies

10. UNIX for Beginners Questions & Answers

Parallel execution of Oracle procedure in UNIX

i have say x number of procedure to run, ie i have one procedure which accepts variable and i need that to run in parallel and capture the error code if in case if it fails through the unix. sqlplus <EOF> exec test_t (abc,124); </EOF> sqlplus <EOF> exec test_t (abc,125); </EOF> sqlplus <EOF>... (2 Replies)
Discussion started by: ATWC
2 Replies
gnt-debug(8)							   Version 2.5.2						      gnt-debug(8)

Name
       gnt-debug - Debug commands

Synopsis
       gnt-debug {command} [arguments...]

DESCRIPTION
The gnt-debug is used for debugging the Ganeti system. COMMANDS
IALLOCATOR iallocator [--debug] [--dir DIRECTION] {--algorithm ALLOCATOR } [--mode MODE] [--mem MEMORY] [--disks DISKS] [--disk-template TEMPLATE] [--nics NICS] [--os-type OS] [--vcpus VCPUS] [--tags TAGS] {instance} Executes a test run of the iallocator framework. The command will build input for a given iallocator script (named with the --algorithm option), and either show this input data (if DIREC- TION is in) or run the iallocator script and show its output (if DIRECTION is out). If the MODE is allocate, then an instance definition is built from the other arguments and sent to the script, otherwise (MODE is relocate) an existing instance name must be passed as the first argument. This build of Ganeti will look for iallocator scripts in the following directories: /usr/local/lib/ganeti/iallocators,/usr/lib/ganeti/ial- locators; for more details about this framework, see the HTML or PDF documentation. DELAY delay [--debug] [--no-master] [-n NODE...] {duration} Run a test opcode (a sleep) on the master and on selected nodes (via an RPC call). This serves no other purpose but to execute a test oper- ation. The -n option can be given multiple times to select the nodes for the RPC call. By default, the delay will also be executed on the master, unless the --no-master option is passed. The delay argument will be interpreted as a floating point number. SUBMIT-JOB submit-job [--verbose] [--timing-stats] [--job-repeat N] [--op-repeat N] {opcodes_file...} This command builds a list of opcodes from files in JSON format and submits a job per file to the master daemon. It can be used to test options that are not available via command line. The verbose option will additionally display the corresponding job IDs and the progress in waiting for the jobs; the timing-stats option will show some overall statistics inluding the number of total opcodes, jobs submitted and time spent in each stage (submit, exec, total). The job-repeat and op-repeat options allow to submit multiple copies of the passed arguments; job-repeat will cause N copies of each job (input file) to be submitted (equivalent to passing the arguments N times) while op-repeat will cause N copies of each of the opcodes in the file to be executed (equivalent to each file containing N copies of the opcodes). TEST-JOBQUEUE test-jobqueue Executes a few tests on the job queue. This command might generate failed jobs deliberately. LOCKS locks [--no-headers] [--separator=SEPARATOR] [-v] [-o [+]FIELD,...] [--interval=SECONDS] Shows a list of locks in the master daemon. The --no-headers option will skip the initial header line. The --separator option takes an argument which denotes what will be used between the output fields. Both these options are to help scripting. The -v option activates verbose mode, which changes the display of special field states (see ganeti(7)). The -o option takes a comma-separated list of output fields. The available fields and their meaning are: mode Mode in which the lock is currently acquired (exclusive or shared) name Lock name owner Current lock owner(s) pending Threads waiting for the lock If the value of the option starts with the character +, the new fields will be added to the default list. This allows one to quickly see the default list plus a few other fields, instead of retyping the entire list of fields. Use --interval to repeat the listing. A delay specified by the option value in seconds is inserted. REPORTING BUGS
Report bugs to project website (http://code.google.com/p/ganeti/) or contact the developers using the Ganeti mailing list (ganeti@google- groups.com). SEE ALSO
Ganeti overview and specifications: ganeti(7) (general overview), ganeti-os-interface(7) (guest OS definitions). Ganeti commands: gnt-cluster(8) (cluster-wide commands), gnt-job(8) (job-related commands), gnt-node(8) (node-related commands), gnt- instance(8) (instance commands), gnt-os(8) (guest OS commands), gnt-group(8) (node group commands), gnt-backup(8) (instance import/export commands), gnt-debug(8) (debug commands). Ganeti daemons: ganeti-watcher(8) (automatic instance restarter), ganeti-cleaner(8) (job queue cleaner), ganeti-noded(8) (node daemon), ganeti-masterd(8) (master daemon), ganeti-rapi(8) (remote API daemon). Ganeti htools: htools(1) (generic binary), hbal(1) (cluster balancer), hspace(1) (capacity calculation), hail(1) (IAllocator plugin), hscan(1) (data gatherer from remote clusters). COPYRIGHT
Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 Google Inc. Permission is granted to copy, distribute and/or modify under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. On Debian systems, the complete text of the GNU General Public License can be found in /usr/share/common-licenses/GPL. Ganeti gnt-debug(8)
All times are GMT -4. The time now is 01:25 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy