Load Balancing in UNIX


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Load Balancing in UNIX
# 8  
Old 02-28-2017
Quote:
Originally Posted by Zaib
I want to start transfer in parallel to multiple hosts and once first 5 files are uploaded to all 5 servers then next chunk onwards.
Create a Korn-Shell-script to do the job. Set up the ftp-jobs as separate functions inside the script and use the "coprocess-facility" to steer them. See the man page for ksh (either ksh88 or ksh93, both can do it) for details about co-processes.

(Note, that other shells lack this special feature, so you will need a real Korn shell to use it, neither bash, pdksh nor similar lookalikes.)

I hope this helps.

bakunin
# 9  
Old 03-01-2017
the following simulates a transfer of the files in the current directory to 5 servers.
Code:
#!/bin/bash
servers=(
 1.1.1.1 2.2.2.2
 3.3.3.3 4.4.4.4
 5.5.5.5
)
# /bin/ksh syntax:
#set -A servers 1.1.1.1 2.2.2.2 3.3.3.3 4.4.4.4 5.5.5.5

sindex=0
smax=${#servers[@]}

for file in *
do
 server=${servers[$sindex]}
 # execute the ( subshell ) in the background
 (
 echo "transfer $file to $server"
 sleep 10
 ) &
 sindex=$(( sindex + 1 ))
 if [ $sindex = $smax ] 
 then
  sindex=0
  wait
 fi
done

This method transfers each file once to one random server.
Replace the echo and sleep commands by a suitable transfer command.

---------- Post updated at 14:24 ---------- Previous update was at 13:43 ----------

Here is a refined version that ensures that always the maximum number of jobs are running.
Instead of waiting for all background jobs to finish it polls the number of jobs in a loop.
Code:
#!/bin/bash
servers=(
 1.1.1.1 2.2.2.2
 3.3.3.3 4.4.4.4
 5.5.5.5
)
# /bin/ksh syntax:
#set -A servers 1.1.1.1 2.2.2.2 3.3.3.3 4.4.4.4 5.5.5.5

sindex=0
smax=${#servers[@]}
maxjobs=$smax

for file in *
do
 # ensure it is a file
 [ -f "$file" ] || continue
 server=${servers[$sindex]}
 # execute the ( subshell ) in the background
 (
 echo "transfer $file to $server"
 sleep $(( RANDOM % 10 ))
 ) &
 sindex=$(( (sindex + 1) % smax ))
 # poll the number of jobs until it becomes less than maxjobs
 until [ `jobs | wc -l` -lt $maxjobs ]
 do
  sleep 2
 done
done

This User Gave Thanks to MadeInGermany For This Post:
# 10  
Old 03-02-2017
Quote:
Originally Posted by MadeInGermany
the following simulates a transfer of the files in the current directory to 5 servers.
Hi,
It is very much helpful. Indeed I learnt few new things as well.
Thanks for help. I am now finalizing it for my requirements.

Once done, will post here so others can also get help.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Load balancing in Autosys

Hi, I am working on development project where I have to migrate many jobs from Tidal to Autosys R11. During this project we came across the following requirements. 1. There are 3 real machines. There could be many jobs activated simultaneously, but only one job should execute at a time and... (0 Replies)
Discussion started by: sujeetp
0 Replies

2. UNIX for Advanced & Expert Users

Help in MQ load balancing

Hi, Currently we have 3 old and 3 new servers catering to Live traffic. As my component move from legacy interfaces to MQ one, we want to have load balancing of old interfaces available on MQ interface as well. For this, we want to send only 30% of all MQ traffic on 3 OLD Live servers, and want... (1 Reply)
Discussion started by: senkerth
1 Replies

3. IP Networking

Load Balancing ppp

Hello everybody How can i Load Balance two slow ppp(gprs) connections with iptables . (4 Replies)
Discussion started by: rink
4 Replies

4. Linux

HTTP load balancing.

Hi, We have 2 pools of servers. Lets call them A and B and they would contain 2 servers each. Pool A will be hosting www.example.com/app/v1 and pool B will be hosting www.example.com/app/v2. Clients will be requesting right url (/v1 or /v2) but will be hitting just one IP. I'd like to: 1)... (3 Replies)
Discussion started by: chrisfb
3 Replies

5. UNIX for Advanced & Expert Users

High availability/Load balancing

Hi folks, (Sorry I don't know what its technology is termed exactly. High Availability OR load balancing) What I'm going to explore is as follows:- For example, on Physical Servers; Server-1 - LAMP, a working server Server-2 - LAMP, for redundancy While Server-1 is working all... (3 Replies)
Discussion started by: satimis
3 Replies

6. Web Development

Load Balancing in Apache

Hi All, I have one webserver which has an application for a set of internal users can be accessed by _http://server1.com I am planning to load balance this application. For that I have cloned this server and build a new one which can be accessed using _http://server2.com]Server2.com. Also i... (2 Replies)
Discussion started by: Tuxidow
2 Replies

7. Solaris

Load balancing with IPMP

Is it possible to do a load balancing ( incoming and outgoing )with with IPMP in solaris 10 like sun trunking ? If yes what are the steps involved in it , i know how to do the failover IPMP both link based and probe based but i 'm looking for possible load balancing (3 Replies)
Discussion started by: fugitive
3 Replies

8. Ubuntu

perlbal and load balancing

Hi guys, I wonder if someone would be able to assist with my problem. I have just set up a load balancer for a company I am working for. HTTP redirection is working fine, however they also want to load balance SSH and FTP too. At the moment the perlbal config looks like; CREATE POOL webhttp ... (1 Reply)
Discussion started by: JayC89
1 Replies

9. AIX

idea on hacmp load balancing

Hi All, Any idea about load balancing on hacmp? Or load balancing is only on lpar. Any idea or link info will do. Thanks in advance. (2 Replies)
Discussion started by: itik
2 Replies

10. UNIX for Dummies Questions & Answers

Question about load balancing

If you have two or more servers load balancing, are the servers mirroring one another? If images, etc., are uploaded, will they be stored on all the servers so that if one server goes down, the images will be served up by another server? (1 Reply)
Discussion started by: wvmlt
1 Replies
Login or Register to Ask a Question