Execute commands parallel in a for loop ?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Execute commands parallel in a for loop ?
# 1  
Old 11-25-2005
Execute commands parallel in a for loop ?

Hi,

please can someone point me in the right direction with a shell scripting problem.
I want to execute a command in a for loop and the command should be started not one-by-one meaning the for loop is waiting for the exit code , it should be started in parallel.

I have a plain text file where a hostname stands in each line.

#!/bin/bash

file="/tmp/serverlist"

for server in `cat $file` ; do

ssh admin@$server "command to run"

done

the problem is :

the command I execute with SSH takes approx. 10 minutes to complete.

I have to run the command on 145 servers and I dont want to wait 10 mins to go to the next. I want to execute it in parallel say on 10 servers at a time.

Any ideas ? simply putting a & after the SSH commands didnt work, nor with continue ?


thanks for your help again...
# 2  
Old 11-25-2005
try this,
with every instance executing command on 10 servers ...
not tested

Code:
>subscript.sh

#!/bin/bash

file=$1
for server in `cat $file` ; do
ssh admin@$server "command to run"
done
/bin/rm $1
exit 0

Code:
>mainscript.sh

#!/bin/bash

file="/tmp/serverlist"
TMPFILE="/tmp/"
cnt=0
filecnt=1
headcnt=10
tailcnt=`cat $file | wc -l`

while [ $cnt -lt $tailcnt ]
do
head -$headcnt $file | tail -10 > $TMPFILE$filecnt
subscript.sh $TMPFILE$filecnt &
cnt=$(($cnt + 10)
headcnt=$(($headcnt + 10))
filecnt=$(($filecnt + 1))
done

exit 0

# 3  
Old 11-25-2005
ssh should be able to work as a background process. Maybe if you enclose the command you want to issue in double quotes to make clear where it ends?

Code:
ssh $user@$host "$command" &

Another gotcha is if the keys are not exchanged ssh stops in an interative mode asking for passwords - really silly programming IMHO - so your script would hang there indefinitely prompting perhaps cron or so for an answer.

bakunin

Last edited by bakunin; 11-25-2005 at 05:02 PM..
# 4  
Old 11-27-2005
put the ssh command in a sub-shell in the background ...

Code:
#!/bin/bash

file="/tmp/serverlist"

for server in `cat $file` ; do

(ssh admin@$server "command to run" < /dev/null &)

done

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parallel RM and FIND commands conflicting

Hi all. Long time!! Hope you're doing well.. I've stumbled on a peculiar siutaion here, and would expect help from this forum on a clean resolution. We are running an rm and find command simultaneously from two different Unix sessions of the same user(let's say USER01) and on the same... (3 Replies)
Discussion started by: kumarjt
3 Replies

2. Shell Programming and Scripting

Parallel increment of nested for loop

Hi, I am using solaris 5.10 environment and need help on doing parallel increment of nested for loop. Samples #inside the code the values assigned to a variable by another awk command will be like a=/xyz/pg/as /xyz/pg/as2 /xyz/pg/as3 b=/xyz/sd/fd1 /xyz/sd/fd2 /xyz/sd/fd3 for q in... (1 Reply)
Discussion started by: ananan
1 Replies

3. Shell Programming and Scripting

Run script in parallel in while loop

Hi I am running a loop which actually runs same script for different argument value passed to it. while read repID do echo "Starting for $repID"; date; perl process_report.pl $repID done<${FILE_TO_READ} However this runs in sequence. I want the loop to not to wait for perl to... (3 Replies)
Discussion started by: dashing201
3 Replies

4. Shell Programming and Scripting

Run the for loop in parallel

I have the below code which runs on multiple databases , but this runs one-after-one. I will need this to run in parallel so that i could save a lot of time. Please help!!! Thanks in advance for Db in `cat /var/opt/oracle/oratab |egrep -v "ASM" |grep -v \# |cut -d\: -f1` do { export... (5 Replies)
Discussion started by: jjoy
5 Replies

5. Shell Programming and Scripting

For loop in parallel

Hello, My script shell is: for i in $(seq $nb_lignes) do //command java done Please, how can i execute all iteration in parallel ? Thank you so much. (9 Replies)
Discussion started by: chercheur857
9 Replies

6. Shell Programming and Scripting

Execute scripts in Parallel

Hi I want to execute few scripts in Parallel. There is a Master Script (MS.ksh) which will call internally all the scripts we need to run in Parallel. Say there are three set of scripts : ABC_1.ksh --> ABC_2.ksh --> ABC_3.ksh (execute ABC_2 when ABC_1 is successful ; Execute ABC_3 only when... (6 Replies)
Discussion started by: dashing201
6 Replies

7. Shell Programming and Scripting

Find and execute shell scripts in multiple sub directories in parallel

I have one parent directory and within that parent directory there are several other sub-directories and within those sub-directories there are several other "large number" of sub-directories. All the sub directories have a shell script in them with a common file name execute_command.sh I want... (4 Replies)
Discussion started by: shoaibjameel123
4 Replies

8. Shell Programming and Scripting

parallel while loop based on the file records

Hi, I need to execute parallel with while loop. Input File(source_file.csv) contains filenames the below source_file.csv file contains Customer1.txt Product1.txt Sales.txt Emp.txt Dept.txt Based on the number of rows that file I want to run the script ‘n' times. while... (2 Replies)
Discussion started by: onesuri
2 Replies

9. Shell Programming and Scripting

Executing two commands in parallel

Hi, I am stuck into a situation where i want to execute a command in my shell script well along with a previous command in order to achieve something but i am not figuring out a way. here is a snippet: service management restart rm -rf lock.file in the above, if you see, i am trying to... (5 Replies)
Discussion started by: sunrexstar
5 Replies

10. Shell Programming and Scripting

Can BASH execute commands on a remote server when the commands are embedded in shell

I want to log into a remote server transfer over a new config and then backup the existing config, replace with the new config. I am not sure if I can do this with BASH scripting. I have set up password less login by adding my public key to authorized_keys file, it works. I am a little... (1 Reply)
Discussion started by: bash_in_my_head
1 Replies
Login or Register to Ask a Question