Code:
Summary of typical IPMP Configurations
1. Production and test interfaces in the same IP subnet
1.1 With defaultrouter
1.2 Without defaultrouter
1.3 With dedicated hosts acting as test targets with "host-routes"
1.4 Configuration example for 1.1, 1.2 and 1.3
2. Production and test interfaces in different IP subnets but the same
physical network
2.1 With defaultrouter in production subnet and test subnet
2.2 With defaultrouter in production subnet but without defaultrouter in test subnet
2.3 With dedicated hosts acting as test targets with "host-routes"
2.4 Configuration example for 2.1, 2.2 and 2.3
2.5 Configuration example for 2.2 if you use IPMP on the test subnet
A. Start script for adding static "host routes" permanently
/etc/init.d/ipmp.targets
B. Summary
Note: This document does not cover the basic knowledge of IP Multipathing. If you would need it, please check it in Sun Microsystems Documentation
All IPMP patches mentioned below should be installed.
Sparc Intel Comments
OS
Release
7 or earlier N/S N/S IPMP was introduced with Solaris[TM] 8 Update 2
for Solaris 8 FCS the Patch ID: 108528,
116965 (109898), 109900 are required
8 108528 108529 Kernel Patch (in.mpathd, ip, icmp, ifconfig)
116965 116966 ip/arp Patch
109900 109901 Startup Patch (/etc/init.d/network and /sbin/ifparse)
109902 109903 Offlining capability (Update 3
110378 110396 Mipagent patch needed for offlining capability
116991 116992 RCM Daemon
9 112233 112234 Kernel Patch (ip)
114344 119435 in.mpathd, arp, ifconfig (see SunAlert 57435)(1)
112912 N/P libinetcfg.so
112854 N/P icmp
113073 114733 ipmp header files
10 118833 118855 Kernel Patch (in.mpathd, ip, icmp, ifconfig)
Key
-----
N/S - Product not supported on this OS
N/P - No patch applicable for this OS
Legend:
-------
IPMP system with IPMP group
T<n> target host <n>
p network/host part of IP address of interface in production subnet
t network/host part of IP address of interface in test subnet
--- data link subnet (i.e broadcast domain) with one IP subnet
=== data link subnet (i.e broadcast domain) with two (or more) IP
subnets
Good to know:
-------------
The operation of IP Multipathing (in.mpathd) depends on the routing
configuration. Therefore in.mpathd always refers to the routing-table
(IRE-cache) to distinguish which test partner(s) are going to be used.
Test partners are required for deciding if the interface is working properly.
in.mpathd by default chooses the defaultrouter as single test-target
for probing. If no defaultrouter exists for the test-interface ip address,
arbitrary hosts on the link are detected by sending out "all hosts"
multicast packets (224.0.0.1) on the wire to detect its test-partners.
An "all routers" multicasts (224.0.0.2) will never be sent! The first five
hosts that are responding to the echo packets are chosen as targets
for probing. In such a non-defaultrouter environment, the in.mpathd
always tries to find five probe targets via an "all hosts" multicast.
The in.mpathd detects failures and repairs by sending out 'icmp-echo'
probes (like pinging) from all interfaces that are part of the IPMP group.
If there are no replies to five consecutive probes, the interface is
considered to have failed and performs a failover of the network access
to another interface in the IPMP group. The probing rate depends on the
failure detection time which is defined in /etc/default/mpathd. By default,
failure detection time is 10 seconds. Thus the five probes will be sent
within the failure detection time.
1. Production and test interfaces in the same IP subnet
======================================================
1.1 With defaultrouter
----------------------
+-------------+
|defaultrouter|
+------o------+
| p=t:172.20.20.1
|
|
----------+-+------------------------+-------------------------
| | p=t:172.20.20/24
p:172.20.20.10 | |
t:172.20.20.210 | | t:172.20.20.220
+---o-o---+
| IPMP |
+---------+
IPMP only use the defaultrouter as probe target. Each test interface of the
IPMP group send ICMP requests only to the defaultrouter. To get the
configuration, IPMP looks to the routing table and is independent from
/etc/defaultrouter file. NO "all hosts" multicast (224.0.0.1) will be sent.
Advantages:
- Easiest configuration for IPMP.
Disadvantages:
- When the defaultrouter is down IPMP failover does not work anymore.
The in.mpathd does NOT send out multicasts to get other probe
targets, therefore all interfaces in the IPMP group get the state
"failed". You can ignore this bug/feature when you have a
defaultrouter which is 100% online! Please look to RFE 4431511 and
4489960 for further information.
- If you have a lot of IPMP groups, the defaultrouter has to reply to
a lot of ICMP requests. Take care of defaultrouter. Do not overload
the defaultrouter.
- The defaultrouter has to reliably answer ICMP echo requests. (e.g.
firewalls sometimes do not)
1.2 Without defaultrouter
-------------------------
+---------+ +---------+
| T1 | ...... | T5 |
+----o----+ +----o----+
| p=t:172.20.20.110 | p=t:172.20.20.150
| |
| |
-----------+-+--------------+---------------------+-------
| | p=t:172.20.20/24
p:172.20.20.10 | |
t:172.20.20.210 | | t:172.20.20.220
+---o-o---+
| IPMP |
+---------+
IPMP dynamically determines five arbitrary hosts on the link via "all
hosts" multicast address (224.0.0.1). At the least, you need one probe
target that IPMP will work. Beware that one probe target is not reliable
enough. If there are less than five targets available the in.mpathd sent
out the "all hosts" multicasts to get a complete list of five probe targets.
Advantages:
- easiest configuration for IPMP in a subnet without a defaultrouter.
- is very reliable due to the five targets.
Disadvantages:
- a subnet without an defaultrouter is very rare.
1.3 With dedicated hosts acting as test targets with "host-routes"
----------------------------------------------------------------
+-------------+ +---------+ +---------+
|defaultrouter| | T1 | ...... | T5 |
+------o------+ +----o----+ +----o----+
! | p=t:172.20.20.110 | p=t:172.20.20.150
! | |
! | |
------------+-+---------------+---------------------+-------
| | p=t:172.20.20/24
p:172.20.20.10 | |
t:172.20.20.210 | | t:172.20.20.220
+---o-o---+
| IPMP |
+---------+
Some "host routes" will be defined with a startscript in
/etc/rc2.d/S70staticroutes. (The script is attached to this document.)
When IPMP refer to the routing table it will choose the first five defined
"host routes" as probe targets. This is due to the fact that normally the
"host routes" are before the defaultrouter in the routing table. If you
have less than five "host routes" also the defaultrouter (when available)
will be used as probe target as well.
Example:
a) Configuration with host1, host2 ... hostN (with N=5 or N>5),
defaultrouter :
==> The first five hosts (host1 ... host5) will be defined as target.
b) Configuration with less than 5 hosts : for instance, host1, host2,
defaultrouter :
==> The three systems (host1, host2, defaultrouter) will be defined as
target.
Also in this case the in.mpathd tries to get five probe targets all
the time from the routing table. Remember in this configuration the
in.mpathd does NOT send "all hosts" multicasts!
Advantages:
- The defaultrouter is not important for the IPMP configuration because
if the defaultrouter is not available you have still some "host routes"
for probing.
- IPMP is always high available due to independency to the defaultrouter
Disadvantages:
- More administrative work to do.
- Due to static configuration you should check that some of the probe
targets are always available.
1.4 Configuration examples for 1.1, 1.2 and 1.3
-------------------------------------------------
/etc/hosts
----------
127.0.0.1 localhost
172.20.20.10 host10 loghost
172.20.20.210 host10-test-qfe0
172.20.20.220 host10-test-qfe4
/etc/hostname.q fe0
------------------
host10 netmask + broadcast + group ipmp0 up \
addif host10-test-qfe0 deprecated -failover netmask + broadcast + up
/etc/hostname.qfe4
------------------
host10-test-qfe4 deprecated -failover netmask + broadcast + group ipmp0 up
ifconfig output:
----------------
qfe0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e8:88:dc
qfe0:1:
flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 3
inet 172.20.20.210 netmask ffffff00 broadcast 172.20.20.255
qfe4:
flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 4
inet 172.20.20.220 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e8:89:34
Note: The example describe the setup of an active-active IPMP configuration
which is best practices. But you can also configure an active-standby
configuration. You simple have to add the "standby" flag to the
/etc/hostname.qfe4 file. More Details available in Technical Instruction < Solution: 217158 >.
2. Production and test interfaces in different IP subnets
==========================================================
If you have not enough additional ip-addresses on hand for setting up
IPMP, you can configure the ipmp-test-interfaces in a different
ip-network than your production network (e.g. 192.168., 10. ..). But you
must make sure that there are enough test-partners (also in the new
test-network) who are responding to the ipmp-test-interfaces. You may
also configure a defaultrouter in the new test-network in case you have
an existing 100.1% reliable test-partner which should act as a single
test-partner. In such a configuration in.mpathd will only use its
test-subnet IP addresses as source address for outgoing probe packets.
Note: The in.mpathd only looks to the test subnet. Therefore if you
have no IP addresses available in the test subnet the IPMP
group will fail although if the production subnet is available.
2.1 With defaultrouter in production subnet and test subnet
-----------------------------------------------------------
+-------------+
|defaultrouter|
+------o------+
| p: 172.20.20.1
| t: 192.168.1.1
|
| t: 192.168.1/24
===========+=+========================+==========================
| | p: 172.20.20/24
p: 172.20.20.10 | |
t: 192.168.1.210 | | t: 192.168.1.220
+---o-o---+
| IPMP |
+---------+
IPMP only use the defaultrouter as probe target. Each test interface of the
IPMP group send ICMP requests only to the defaultrouter. To get the
configuration IPMP looks to the routing table and is independent from
/etc/defaultrouter file. NO "all hosts" multicast (224.0.0.1) will be sent.
Advantages:
- Test interfaces don't need IP addresses of the production subnet
Disadvantages:
- The interface of the defaultrouter has to reside in both the
production AND test subnet.
- Exceptional configuration of defaultrouter.
- All which are mentioned in section 1.1
2.2 With defaultrouter in production subnet net but without defaultrouter
in test subnet
----------------------------------------------------------------------------------------
+-------------+ +---------+ +---------+
|defaultrouter| | T1 | ... | T5 |
+------o------+ +----o----+ +----o----+
| p:172.20.20.1 | p:172.20.20.110 | p:172.20.20.150
| | t:192.168.1.110 | t:192.168.1.150
| | |
| | | t:192.168.1/24
============+=+=+==============+=================+=====================
| | p:172.20.20/24
p:172.20.20.10 | |
t:192.168.1.210 | | t:192.168.1.220
+---o-o---+
| IPMP |
+---------+
IPMP dynamically determines five arbitrary hosts in the test subnet via
"all hosts" multicast address (224.0.0.1). At least you need one probe
target that IPMP will work. Beware that one probe target is not reliable
enough. If there are less than five targets available the in.mpathd sent
out the "all hosts" multicasts to get a complete list of five probe
targets.
Advantages:
- easiest configuration for IPMP if you have too less IP addresses
available in the production subnet.
- is very reliable due to the five targets.
Disadvantages:
- the probe targets must be available before you can setup the
IPMP host.
- more administrative work because you have to setup the probe
targets as an additional interface in the test subnet.
Recommendation: Setup an additional logical network interface on target host.
(e.g. add a new interface to /etc/hostname.qfe0 with the 'addif' option)
If your test partners are on systems which run also IPMP you must add an
logical network interface NOT flagged deprecated and NOT flagged nofailover.
An address associated with this interface will then be used as source address
by responding properly to "all hosts" multicast packets which are used for
the automatic IPMP test partner detection.
Beware that Solaris 8 does NOT require the additional logical network
interface in the test subnet on the target host. So, in case of an upgrade
from Solaris 8 to Solaris[TM] 9 or higher you have to change your configuration.
Please refer to 2.5 for a detailed example.
2.3 with dedicated hosts acting as test targets with "host-routes"
-----------------------------------------------------------------
+-------------+ +---------+ +---------+
|defaultrouter| | T1 | .... | T5 |
+------o------+ +----o----+ +----o----+
! | p:172.20.20.110 | p:172.20.20.150
! | t:192.168.1.110 | t:192.168.1.150
! | |
! | | t:192.168.1/24
=============+=+===============+=================+=====================
| | p:172.20.20/24
p:172.20.20.10 | |
t:192.168.1.210 | | t:192.168.1.220
+---o-o---+
| IPMP |
+---------+
Some "host routes" will be defined in the test subnet with a startscript in
/etc/rc2.d/S70ipmp.targets (The script is attached to this document.). When
IPMP refer to the routing table it will choose the first five defined "host
routes" as probe targets in the test subnet. This is due to the fact that
normally the "host routes" are before the defaultrouter in the routing
table. If you have less than five "host routes" also the defaultrouter
(when available in the test subnet) will be used as probe target as well.
Example:
- please look to the example of section 1.3
Advantages:
- test interfaces don't need IP addresses of the production subnet
- all which are mentioned in section 1.3
Disadvantages:
- all which are mentioned in section 1.3
- all which are mentioned in section 2.2
2.4 Configuration examples for 2.1, 2.2 and 2.3
------------------------------------------------
/etc/hosts
----------
127.0.0.1 localhost
172.20.20.10 host10 loghost
192.168.1.210 host10-test-qfe0
192.168.1.220 host10-test-qfe4
/etc/hostname.qfe0
------------------
host10 netmask + broadcast + group ipmp0 up \
addif host10-test-qfe0 deprecated -failover netmask + broadcast + up
/etc/hostname.qfe4
------------------
host10-test-qfe4 deprecated -failover netmask + broadcast + group ipmp0 up
ifconfig output:
----------------
qfe0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e8:88:dc
qfe0:1:
flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 3
inet 192.168.1.210 netmask ffffff00 broadcast 192.168.1.255
qfe4:
flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 4
inet 192.168.1.220 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp0
ether 8:0:20:e8:89:34
Note: The example describe the setup of an active-active IPMP configuration
which is best practices. But you can also configure an active-standby
configuration. You simple have to add the "standby" flag to the
/etc/hostname.qfe4 file. More Details available in Technical Instruction < Solution: 217158 >.
2.5 Configuration example for 2.2 if you use IPMP on the test subnet
---------------------------------------------------------------------
/etc/hosts
----------
127.0.0.1 localhost
# IPMP active IP address:
172.20.20.10 host10 loghost
# IPMP test interface
192.168.1.210 host10-test-ce0
# additional active interface in the 'test subnet'
# to be able to respond to probe packets from other
# IPMP setups (See 2.2). Will also failover.
192.168.1.211 host10-prod-ce0
# IPMP test interface
192.168.1.220 host10-test-ce1
/etc/hostname.ce0
-----------------
172.20.20.10 netmask + broadcast + group ipmp0 up \
addif 192.168.1.211 netmask + broadcast + up \
addif 192.168.1.210 netmask + broadcast + deprecated -failover up
/etc/hostname.ce1
-----------------
192.168.1.220 netmask + broadcast + group ipmp0 deprecated -failover up
ifconfig output:
----------------
ce0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e2:da:d5
ce0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.1.211 netmask ffffff00 broadcast 192.168.1.255
ce0:2: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
inet 192.168.1.210 netmask ffffff00 broadcast 192.168.1.255
ce1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 192.168.1.220 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp0
ether 8:0:20:e2:da:d6
A.
----------- Begin of start script /etc/init.d/ipmp.targets --------------
#!/sbin/sh
# /etc/rc2.d/S70ipmp.targets /etc/init.d/ipmp.targets
# Copyright (c) 2005 by Sun Microsystems, Inc.
# All rights reserved.
#
#ident "@(#)ipmp.targets 1.0.0
#
# Edit the following IPMP test TARGETS to suit your needs.
# To install:
# 1) cp ipmp.targets /etc/init.d
# 2) perform edits on the scri pt as required (e.g: add TARGETS)
# 3) chmod 744 /etc/init.d/ipmp.targets
# 4) chown root:sys /etc/init.d/ipmp.targets
# 5) ln /etc/init.d/ipmp.targets /etc/rc2.d/S70ipmp.targets
#
TARGETS="192.168.85.117 192.168.85.127 192.168.85.137"
case "$1" in
'start')
/usr/bin/echo "Adding static routes for use as IPMP targets"
for target in $TARGETS; do
/usr/sbin/route add -host $target $target
done
;;
'stop')
/usr/bin/echo "Removing static routes for use as IPMP targets"
for target in $TARGETS; do
/usr/sbin/route delete -host $target $target
done
;;
esac
----------- End of start script /etc/init.d/ipmp.targets --------------
See also: How to Configure a BOOTP Relay Agent (DHCP Manager) (System Administration Guide: IP Services) - Sun Microsystems
B. Summary
It's not easy to give a general recommendation because it depends on the
network infrastructure which the customers have. Therefore you have to
discuss the various possibilities with your customer. Maybe the most used
setups are 1.1 and 2.2.
