Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

Changing single path NIC to a teamed connection in same subnet

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Changing single path NIC to a teamed connection in same subnet
# 1  
Old 09-16-2019
Changing single path NIC to a teamed connection in same subnet
Dear all,

I have a remote CentOS7 server that has two network cards. Each card has four ports and port one of card one was defined with the IP address assigned to the server. So far, so good and it's been working for over a year. We have now got cables sorted out so there are four paths available (two from each card, via two switches) to give us resilience. After that, I have built a logical teamed device with the three new paths and it's all working just fine until I try to turn off the old un-teamed/single path eno1. I think it's all down to routing and I'm stuck.

Output from ip add show (removed unused ports for clarity):-
Code:
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b8:83:13:55:10:e0 brd ff:ff:ff:ff:ff:ff
    inet 10.102.16.11/24 brd 10.102.16.255 scope global noprefixroute eno1
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master team0 state UP group default qlen 1000
    link/ether 98:f2:b3:1d:3e:04 brd ff:ff:ff:ff:ff:ff
6: eno5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master team0 state UP group default qlen 1000
    link/ether 98:f2:b3:1d:3e:04 brd ff:ff:ff:ff:ff:ff
7: eno6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master team0 state UP group default qlen 1000
    link/ether 98:f2:b3:1d:3e:04 brd ff:ff:ff:ff:ff:ff
22: team0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 98:f2:b3:1d:3e:04 brd ff:ff:ff:ff:ff:ff
    inet 10.102.16.13/24 brd 10.102.16.255 scope global noprefixroute team0
       valid_lft forever preferred_lft forever

All seems well and I have changed the DNS to refer to the new IP address for the team0 device. I can SSH to the new IP address and verify with netstat that I am connected to the new IP. Output from netstat -nr and route -n:-
Code:
# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.102.16.254   0.0.0.0         UG        0 0          0 eno1
0.0.0.0         10.102.16.254   0.0.0.0         UG        0 0          0 team0
10.102.16.0     0.0.0.0         255.255.255.0   U         0 0          0 eno1
10.102.16.0     0.0.0.0         255.255.255.0   U         0 0          0 team0

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.102.16.254   0.0.0.0         UG    0      0        0 eno1
0.0.0.0         10.102.16.254   0.0.0.0         UG    350    0        0 team0
10.102.16.0     0.0.0.0         255.255.255.0   U     100    0        0 eno1
10.102.16.0     0.0.0.0         255.255.255.0   U     350    0        0 team0
#

If I try to disable eno1 I get no network response until I turn it back on again via the console. It's the same it I delete the default route with route delete default gw 10.102.16.254 dev eno1

I just don't get it. There appears (to me) to be a valid route from device team0 so what am I missing? Smilie

I do see this oddity though:-
Code:
# arp -a
? (10.102.16.254) at <incomplete> on team0
? (10.102.16.254) at 00:22:bd:fa:09:ff [ether] on eno1

For completeness, my server reports the below:-
Code:
# uname -a
Linux my_host 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

After I retire eno1 as a discreet connection, it will be added into the team to give the proper resilience.


I am quite expecting to be shown that I am a fool. I'd prefer that than being left hanging. Smilie

Many thanks, in advance,
Robin
Last edited by rbatte1; 09-16-2019 at 10:19 AM..
# 2  
Old 09-16-2019
What happens when you look at the arp tables, before and after?

On ethernet, as you know, the actual LAN routing is done by MAC address; so when we change IP address, we need to flush / rebuild the arp table (on both ends), generally speaking.

If we change an IP address and do not flush the arp table, there will often be an incorrect IP address <---> MAC address in the arp table and this can cause LAN routing snafus.

PS: I'm writing this based on knowledge from decades ago, off the top of my head, and have not done any research on arp in a long time.
# 3  
Old 09-16-2019
Are you using network manager and nmcli or ?

Regards
Peasant.
# 4  
Old 09-17-2019
Creation was with nmcli yes, although I think we have Network Manager service turned off. The course I went on suggested it doesn't work Smilie

This might actually be a network team thing, as in the switches are not happy seeing the same MAC address & IP address on multiple ports. I will keep looking at that and post an update of how we get on.



Robin
# 5  
Old 10-08-2019
This appears to be a conflict between the server and the network switches all along. We have redefined them as LACP balanced connections and later the network team have done the same to the switch ports. This broke everything because there was still the default route going out on the old card which had been configured by the network team to be in the team. I removed the default route from that port and hey-presto, I've got on to the new IP address properly with data being returned so a proper connection could be established.

I have then reconfigured the now redundant link to be part of the team and restarted the network services. I now have multiple LACP balanced active links.



For anyone else who may find this thread, I have two cards with four ports, so eight possible eno interfaces. Only 1, 2, 5 & 6 are cabled, so the full commands I used to bond them all together are:-
Code:
nmcli con add type team con-name team0 ifname team0 config '{"runner": {"name": "loadbalance", "tx_hash": ["eth", "ipv4", "ipv6"], "tx_balancer": {"name": "basic"} } }'
nmcli con mod team0 ipv4.addresses '10.102.16.13/24' ipv4.method manual
nmcli con add type team-slave ifname eno2 con-name team0-eno2 master team0
nmcli con add type team-slave ifname eno5 con-name team0-eno5 master team0
nmcli con add type team-slave ifname eno6 con-name team0-eno6 master team0

A new MAC address is created for the team and each bonded interface gets the same MAC address.

After the original connection was redundant, I added it to the group with this command and this edit:-
Code:
nmcli con add type team-slave ifname eno1 con-name team0-eno1 master team0
sed -i 's/ONBOOT="yes"/ONBOOT="no"/'  /etc/sysconfig/network-scripts/ifcfg-eno1

Verification commands:
Code:
ip ad s
teamdctl team0 state                        (and variations from this)
nmcli -p
ethtool -S eno1                             (or eno2, eno5 & eno6)


I hope that this is useful to someone, but at least I have it documented for myself too! Smilie


Kind regards,
Robin
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Linux

Add two different subnet public IPs to single NIC or two different NIC on same box

Hello Admins, My ask is how can I add two different subnet IPs to same box with two different gateways? The issue is I can connect to the box when I am on ethernet LAN, but I am not able to connect to the same IP when I am on wifi. The server is RHEL 7 VM on vmware. How can I get connected... (4 Replies)
Discussion started by: snchaudhari2
4 Replies

2. AIX

Changing VLAN on AIX lpars in the same subnet

Hi Guys, Our lpars is currently running on 2 different vlans (20, 30). Now we have a requirement that vlan 30 needs to be change to vlan 31 at the same subnet. I'm not sure on what is the best approach for this or what change is involve on the AIX side. This is our setup. Network switch -... (5 Replies)
Discussion started by: kaelu26
5 Replies

3. Solaris

Solaris 2nd nic for different subnet

posted this on the other sub-forum https://www.unix.com/unix-for-dummies-questions-and-answers/246504-solaris-11-2nd-nic-different-subnet.html apologies if linking is not appropriate Thanks for your help. (0 Replies)
Discussion started by: ad101
0 Replies

4. Solaris

Solaris 11 2nd nic for different subnet

I have a Solaris 11 machine with 2 network cables attached. The first one is the default route and is working okay. I am trying to activate the second on another subnet, but am sure missing something. The first one is on 10.30.128. and with it everything works fine, but when I try to add the... (0 Replies)
Discussion started by: ad101
0 Replies

5. Solaris

Changing the subnet mask on solaris 10

Hello, can anyone help me with the command to change the subnet mask on solaris 10? The mask is currently 255.255.255.255 I will like to change it to 255.255.2555.0. Thank you (5 Replies)
Discussion started by: cjashu
5 Replies

6. AIX

Changing subnet mask for a NIM Network.

I am new to this forum so please bare with me. I did search for this answer prior to posting but no luck. Running an AIX NIM Master at 5.3TL9SP4, with about 100 clients. The subnet of one of the networks defined in the NIM env has changed. When I go to Manage Networks, Change/Show... (0 Replies)
Discussion started by: juredd1
0 Replies

7. Solaris

Need help in changing the subnet mask

Hi, I have a task to edit the subnet mask in almost 100+ solaris servers.Few of the servers are configured with IPMP.There will be no change of IP address or default gateway.If its a single IP we can bring NIC down,change the subnet mask in /etc/netmasks,then bring the NIC back to normal.But... (3 Replies)
Discussion started by: rajip23
3 Replies

8. IP Networking

Migrating existing Subnet to a new subnet and changing ip addresses in UNIX

Hi, My project needs to migrate the existing Subnet (255.255.255.0) to a new subnet and change the ipaddresses (currently C class). How can I do that. I need some information. John (0 Replies)
Discussion started by: johnmarsh
0 Replies

9. IP Networking

changing the subnet mask permanently

I understand how to change the ip address permanently however, I need to also make a permanent change to the subnet mask. How would I accomplish this. (5 Replies)
Discussion started by: johnparksjr
5 Replies

10. IP Networking

DHCPD, Multiple interfaces, Single Subnet

I have an OpenBSD 3.7 firewall with five network interfaces on it, one of which is connected to the Internet. I'd like to use the remaining four interfaces as a network switch for a single internal subnet. The main problem I have is that the DHCP daemon doesn't like multiple interfaces matching... (0 Replies)
Discussion started by: vertigo23
0 Replies
Login or Register to Ask a Question