Sponsored Content
Special Forums UNIX and Linux Applications High Performance Computing Parallel Execution on Multiple System Post 302284672 by 123an on Friday 6th of February 2009 04:53:55 AM
Old 02-06-2009
Parallel Execution on Multiple System

Hi All,

I am working on a project where I need to execute set of arguments (around 500) on a Simulator. If I execute this on one linux (RedHat 8.0) machine it will approximately takes 2~3 days. Since I am having more linux machines am thinking of executing these on different machines in parallel(i.e. Paralle Execution).

I tried for sanity basis following option which is already posted in this forum;
i.e. Script1.sh & Script2.sh & Script3.sh . But I didn't find much savings in this option.

I also tried downloading a tool-JOBQUEUE from site "Download jobqueue 0.04 for Linux - jobqueue is a program for executing jobs in parallel to complete all jobs as fast as possible. - Softpedia".
This is able to split execution on multiple system, but it won't take all arguments list set (like if arguments set list has 10, it will take 8/9 skips 1 r 2).

Could you please help me in solving the problem.

Thanks in advance

Regards,
123an
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parallel Execution

Hello, I wish to run parallel process forked from one script. Currently I doing is submit them in background. For example: --------------------------------------------------------------- #!/usr/bin/ksh process1 & process2 & process3 & ..... ..... #here I check for completion of... (4 Replies)
Discussion started by: RishiPahuja
4 Replies

2. AIX

Make : parallel execution

Hi, Is there any way to run parallel jobs using make command? I am using non-GNU make utility on AIX 4.3. I want to run 2 jobs symultaneously using make utility. Thanks. Suman (0 Replies)
Discussion started by: suman_jakkula
0 Replies

3. Shell Programming and Scripting

Parallel Job Execution

Hi All, At present I am using a UNIX Script which is running a set of JOBS. These JOBS are to be repeated for 20 times, means the same set of JOBS are repeated by just passing different arguments (From 1 to 20). Is there any way by which I can execute them parallel? At present its all... (4 Replies)
Discussion started by: Prashantckc
4 Replies

4. Shell Programming and Scripting

Conditional execution and parallel jobs

how can i process jobs parallel with conditions below. Script1.ksh Script2.ksh Script3.ksh Script4.ksh Script5.ksh Script6.ksh Script7.ksh Script8.ksh Script9.ksh Script10.ksh After successful completion of Script1.ksh I need to run Script7.ksh. After successful... (4 Replies)
Discussion started by: ford2020
4 Replies

5. Shell Programming and Scripting

Parallel execution of script not syncronize

I am haveing 2 scripts, 1st script calls 2ed script for each parameter.(parameters are kept in a different txt file) 1st script for x in `cat Export_Tables_List.txt` do sh Exp_Table.sh $x & done echo -e "1) following tables are successfully exported : \n" > temp cat... (1 Reply)
Discussion started by: sbmk_design
1 Replies

6. Programming

Parallel Execution of Programs

Since there've been a few requests for a method to execute commands on multiple CPUs (logical or physical), with various levels of shell-, make-, or Perl-based solutions, ranging from well-done to well-meant, and mostly specific to a certain problem, I've started to write a C-based solution... (4 Replies)
Discussion started by: pludi
4 Replies

7. UNIX for Advanced & Expert Users

Parallel Execution of Command

Hi All, We have a table that has to store around 80-100 million records. The table is partitioned by a column called Market Code. There are 30 partitions each corresponding to one MRKT_CD. The source of this table is a join between 3-4 other tables. We are loading this table through SQLPLUS... (2 Replies)
Discussion started by: jerome_rajan
2 Replies

8. Shell Programming and Scripting

How to make parallel execution on folder ?

I have few very huge files ~ 2 Billion rows of 130 column(CDR data) in a folder, I have written shell script need to read on each file in a folder and will create a new files based on some logic. But problem is it's taking time to create a new file due to the size , So i dont want to corrupt... (6 Replies)
Discussion started by: rspwilliam
6 Replies

9. Homework & Coursework Questions

Parallel execution on multiple servers in UNIX

I have a requirement (in a shell script) to connect to several servers at once and execute a series of commands. I am aware that ssh can be used for sequential execution. But since most of the commands that I need to execute take a long time, I have to go for the parallel option. Is there... (2 Replies)
Discussion started by: sneha1887
2 Replies

10. UNIX for Beginners Questions & Answers

Parallel execution of Oracle procedure in UNIX

i have say x number of procedure to run, ie i have one procedure which accepts variable and i need that to run in parallel and capture the error code if in case if it fails through the unix. sqlplus <EOF> exec test_t (abc,124); </EOF> sqlplus <EOF> exec test_t (abc,125); </EOF> sqlplus <EOF>... (2 Replies)
Discussion started by: ATWC
2 Replies
condor_findhost(1)					      General Commands Manual						condor_findhost(1)

Name
       condor_findhost find - machine(s) in the pool that can be used with minimal impact on currently running Condor jobs and best meet any spec-
       ified constraints

Synopsis
       condor_findhost [-help] [-m] [-n num] [-c c_expr] [-r r_expr] [-p centralmanagerhostname]

Description
       condor_findhostsearches a Condor pool of machines for the best machine or machines that will have the minimum impact on running Condor jobs
       if  the machine or machines are taken out of the pool. The search may be limited to the machine or machines that match a set of constraints
       and rank expression.

       condor_findhostreturns a fully-qualified domain name for each machine. The search is limited (constrained) to a specific  set  of  machines
       using  the  -coption. The search can use the -roption for rank, the criterion used for selecting a machine or machines from the constrained
       list.

Options
       -help

	  Display usage information and exit

       -m

	  Only search for entire machines. Slots within an entire machine are not considered.

       -n num

	  Find and list up to nummachines that fulfill the specification. numis an integer greater than zero.

       -c c_expr

	  Constrain the search to only consider machines that result from the evaluation of c_expr. c_expris a ClassAd expression.

       -r r_expr

	  r_expris the rank expression evaluated to use as a basis for machine selection. r_expris a ClassAd expression.

       -p centralmanagerhostname

	  Specify the pool to be searched by giving the central manager's host name. Without this option, the current pool is searched.

General Remarks
       condor_findhostis used to locate a machine within a pool that can be taken out of the pool with the least disturbance of the pool.

       An administrator should set preemption requirements for the Condor pool. The expression

       (Interactive =?= TRUE )

       will let condor_findhostknow that it can claim a machine even if Condor would not normally preempt a job running on that machine.

Exit Status
       The exit status of condor_findhostis zero on success. If not able to identify as many machines as requested, it returns one more  than  the
       number  of  machines identified. For example, if 8 machines are requested, and condor_findhostonly locates 6, the exit status will be 7. If
       not able to locate any machines, or an error is encountered, condor_findhostwill return the value 1.

Examples
       To find and list four machines, preferring those with the highest mips (on Drystone benchmark) rating:

       condor_findhost	-n 4 -r "mips"

       To find and list 24 machines, considering only those where the  kflops attribute is not defined:

       condor_findhost	-n 24 -c "kflops=?=undefined"

Author
       Condor Team, University of Wisconsin-Madison

Copyright
       Copyright (C) 1990-2012 Condor Team, Computer Sciences Department, University of  Wisconsin-Madison,  Madison,  WI.  All  Rights  Reserved.
       Licensed under the Apache License, Version 2.0.

       See the Condor Version 7.8.2 Manualor http://www.condorproject.org/licensefor additional notices.

       condor-admin@cs.wisc.edu

								  September 2012						condor_findhost(1)
All times are GMT -4. The time now is 09:12 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy