Sponsored Content
Special Forums UNIX and Linux Applications High Performance Computing Parallel Execution on Multiple System Post 302285855 by 123an on Monday 9th of February 2009 10:58:42 PM
Old 02-09-2009
Quote:
Originally Posted by otheus
With GNU xargs, you can use the -P option to run jobs in parallel. So
Code:
cat input.txt | xargs -n 1 -P 4 myprogram

would run your program with each line of input provided as the arguments to myprogram, and 4 jobs would be started at a time.

To distribute among several Linux nodes, you can build a cluster, as Neo suggested.

OR you can use ssh/rsh to distribute run the jobs on the other hosts. First, make a file containing the hostnames of your cluster. If a host has 2 CPUs, have the hostname entry twice. If it has 4 cores, have 4 entries of that hostname.

From here you can go in different directions, but ultimately, you run one rsh/ssh process her line in this hosts file.
Hi Neo and otheus,

Thanks for ur valuable replies....

I am currently exploring all options suggested here....


Thanks...

123an
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parallel Execution

Hello, I wish to run parallel process forked from one script. Currently I doing is submit them in background. For example: --------------------------------------------------------------- #!/usr/bin/ksh process1 & process2 & process3 & ..... ..... #here I check for completion of... (4 Replies)
Discussion started by: RishiPahuja
4 Replies

2. AIX

Make : parallel execution

Hi, Is there any way to run parallel jobs using make command? I am using non-GNU make utility on AIX 4.3. I want to run 2 jobs symultaneously using make utility. Thanks. Suman (0 Replies)
Discussion started by: suman_jakkula
0 Replies

3. Shell Programming and Scripting

Parallel Job Execution

Hi All, At present I am using a UNIX Script which is running a set of JOBS. These JOBS are to be repeated for 20 times, means the same set of JOBS are repeated by just passing different arguments (From 1 to 20). Is there any way by which I can execute them parallel? At present its all... (4 Replies)
Discussion started by: Prashantckc
4 Replies

4. Shell Programming and Scripting

Conditional execution and parallel jobs

how can i process jobs parallel with conditions below. Script1.ksh Script2.ksh Script3.ksh Script4.ksh Script5.ksh Script6.ksh Script7.ksh Script8.ksh Script9.ksh Script10.ksh After successful completion of Script1.ksh I need to run Script7.ksh. After successful... (4 Replies)
Discussion started by: ford2020
4 Replies

5. Shell Programming and Scripting

Parallel execution of script not syncronize

I am haveing 2 scripts, 1st script calls 2ed script for each parameter.(parameters are kept in a different txt file) 1st script for x in `cat Export_Tables_List.txt` do sh Exp_Table.sh $x & done echo -e "1) following tables are successfully exported : \n" > temp cat... (1 Reply)
Discussion started by: sbmk_design
1 Replies

6. Programming

Parallel Execution of Programs

Since there've been a few requests for a method to execute commands on multiple CPUs (logical or physical), with various levels of shell-, make-, or Perl-based solutions, ranging from well-done to well-meant, and mostly specific to a certain problem, I've started to write a C-based solution... (4 Replies)
Discussion started by: pludi
4 Replies

7. UNIX for Advanced & Expert Users

Parallel Execution of Command

Hi All, We have a table that has to store around 80-100 million records. The table is partitioned by a column called Market Code. There are 30 partitions each corresponding to one MRKT_CD. The source of this table is a join between 3-4 other tables. We are loading this table through SQLPLUS... (2 Replies)
Discussion started by: jerome_rajan
2 Replies

8. Shell Programming and Scripting

How to make parallel execution on folder ?

I have few very huge files ~ 2 Billion rows of 130 column(CDR data) in a folder, I have written shell script need to read on each file in a folder and will create a new files based on some logic. But problem is it's taking time to create a new file due to the size , So i dont want to corrupt... (6 Replies)
Discussion started by: rspwilliam
6 Replies

9. Homework & Coursework Questions

Parallel execution on multiple servers in UNIX

I have a requirement (in a shell script) to connect to several servers at once and execute a series of commands. I am aware that ssh can be used for sequential execution. But since most of the commands that I need to execute take a long time, I have to go for the parallel option. Is there... (2 Replies)
Discussion started by: sneha1887
2 Replies

10. UNIX for Beginners Questions & Answers

Parallel execution of Oracle procedure in UNIX

i have say x number of procedure to run, ie i have one procedure which accepts variable and i need that to run in parallel and capture the error code if in case if it fails through the unix. sqlplus <EOF> exec test_t (abc,124); </EOF> sqlplus <EOF> exec test_t (abc,125); </EOF> sqlplus <EOF>... (2 Replies)
Discussion started by: ATWC
2 Replies
queuedefs(4)						     Kernel Interfaces Manual						      queuedefs(4)

NAME
queuedefs - queue description file for at, batch, and crontab SYNOPSIS
DESCRIPTION
The file describes the characteristics of the queues managed by (see cron(1M)). Each non-comment line in this file describes one queue. The format of the lines are as follows: [njob[nice[nwait The fields in this line are: q The name of the queue, such that is the default queue for jobs started by (see at(1)), is the queue for jobs started by (see at(1)), and is the queue for jobs run from a file (see crontab(1)). Queue names through designate user-defined queues. njob The maximum number of jobs that can be run simultaneously in that queue. Although any number can be specified here, (see cron(1M)) by default limits the number of jobs that can be run on all the queues to 100. This limitation can be removed by setting the variable to 1 in the file. nice The value to give to all jobs in that queue that are not run with a user ID of super-user (see nice(1)). The default value is 2. nwait The number of seconds to wait before rescheduling a job that was deferred because more than njob jobs were running in that job's queue, or because more than 100 jobs were running in all the queues (see njob above). EXAMPLES
Consider the following file: The file is interpreted as follows: The queue, for jobs (see at(1)), can have up to 4 jobs running simultaneously, and those jobs will be run with a value of 1. Since no nwait value is given, if a job cannot be run because too many other jobs are running, will wait 60 seconds before trying again to run it (see cron(1M)). The queue, for jobs (see at(1)), can have up to 2 jobs running simultaneously. Those jobs will be run with a value of 2. If a job cannot be run because too many other jobs are running, will wait 90 seconds before trying again to run it. All other queues can have up to 100 jobs running simultaneously. They will be run with a value of 2, and if a job cannot be run because too many other jobs are running, will wait 60 seconds before trying again to run it. SEE ALSO
at(1), nice(1), crontab(1), cron(1M), proto(4). STANDARDS CONFORMANCE
queuedefs(4)
All times are GMT -4. The time now is 11:48 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy