UNIX for Advanced & Expert Users

Hey everyone. We're in the process of deploying a number of servers. They're HP DL360 G6's. OS is RHEL 5.4

They've got two on board NIC's, as well as an additional dual port network card, giving us a total of 4 NIC's. Eth0 and Eth2 are bonded together using the bonding module, and the resulting interface is called bond0. That works just fine. The default gateway is located in bond0's subnet, so all connectivity works just fine.
For now let's say this is the network information for bond0:
bond0 ip: 192.168.1.10
bond0 DGW: 192.168.1.1
bond0 SNM: 255.255.255.0

Eth1 is on a dedicated management subnet. It doesn't seem to have any connectivity to anything; not even to nodes on its own subnet. Here's some play network info for eth1:
eth1 ip: 192.168.2.10
eth1 SNM: 255.255.255.0
eth1 DGW: 192.168.2.1 (not actually set up anywhere in the system since only one DGW can be defined)

Basically bond0 needs to be able to talk to all hosts in its subnet, as well as another subnet. It should *NEVER* be used for management traffic. Likewise, eth1 should only be used for management traffic on its own subnet. It should *NEVER* be used for the traffic that would normally be on bond0.

I have verified that IP replay is disabled in /etc/sysctl.
Here was my first attempt at setting up routing:

Code:

#!/bin/bash
#Set up the first subnet's routing table (we'll name it 70)
/sbin/ip route flush table 70
/sbin/ip route add table 70 to 192.168.1.0/24 dev bond0
/sbin/ip route add table 70 to default via 192.168.1.1 dev bond0

#Set up the second subnet's routing table (we'll call it 80)
/sbin/ip route flush table 80
/sbin/ip route add table 80 to 192.168.2.0/24 dev eth1
/sbin/ip route add table 80 to default via 192.168.2.1 dev eth1

#Create the rules to choose what table to use. Choose based on source IP
#We need to give the rules different priorities; for convenience name priority
#after the table
/sbin/ip rule add from 192.168.1.0/24 table 70 priority 70
/sbin/ip rule add from 192.168.2.0/24 table 80 priority 80

#Flush the cache to make effective
/sbin/ip route flush cache

It didn't work. When trying to ping it showed all traffic from eth1 going out the IP of bond0, and then gave a "destination host unreachable" error. Not good.
I then started back over from scratch and did the following:

Code:

ip route add to 192.168.2.0/24 dev eth1 table 80
ip rule add from 192.168.2.10 table 80 priority 80
ip rule add to 192.168.2.0/24 table 80 priority 81
ip route add to all dev eth1 table 80
ip route flush cache

This time it shows that the traffic is going out of the IP for eth1, but I'm not sure if its actually leaving this interface or not. Either way, i'm still getting "destination host unreachable" errors even though I know for a fact that the destination host is alive (i'm ssh'd into it right now). I haven't done a tcpdump yet to verify one way or another.

To be patently honest I have no idea what I'm doing right or wrong with this one. The ip rule documentation is spattered with examples but none seem to work for what i'm doing. I still can't even ping a host on a local subnet, even though I've verified that the network settings for eth1 are configured correctly (besides routing).

Any help would be greatly appreciated. Here are the examples I've been using:
RedWolf Internet Linux Multihoming / iproute2 policy routing
10.4.*Multiple Connections to the Internet
Linux Dual Network Route Hack (Multi-Homing) | Sivel.net

Thanks for any light you can help shed.
-Matty

PS: Any 'for idiots' guides to the IPROUTE2 suite would be greatly appreciated. I've read through the documentation and its ancient.

PPS: This is simply a first step, ultimately I'd like to sort which types of traffic go over which interface based on protocol, but I figure that'll be a ton easier once I have the basic gist of things.

---------- Post updated at 03:58 PM ---------- Previous update was at 03:39 PM ----------

As an addendum, just tried running tcpdump. First I ran it on eth1, and tried pinging the eth1 ip from the remote host. No packets received. Tried it again on bond0, pinging the eth1 ip, and again no packets received. So I'm kind of at a loss now.

Prior issues have been resolved. I used the method detailed here to get the interfaces up and running:
Routing for multiple uplinks/providers

All traffic appears to be working correctly; what comes in on one port leaves the correct port. Well, save for one thing. SNMP isn't working correctly. When an SNMP GET request comes in on eth1, the SNMP agent still appears to be sending out responses on bond0, so they're never retrieved by the manager. Any ideas??

What is the IP address of the manager? (i.e. the server that issues the SNMP GET) ?