This is a general question relating to the building of VIO servers;
It has been a long time since I have done any serious AIX work, AIX 4.3 was new and I was for the most part working with 3.2.2 on older RS6000 kit.
So to the current day, I'm about to take delivery of some new tin which I have to plan, install and configure the first batch of tin will comprise.
Three Power 8 S824's
One HMC
One SVC
One EMC VNX5800 Disk Array
The S824's are customer install and will each come with 4 x Dual Port 16Gb HBA's and 4 x Quad Port 10Gb NIC's along with the 4 x 1Gb copper NIC.
So in simple terms my question is:- When building the VIO's can I assign LAN and SAN resource on a port by port basis, or do I have to assign a it on a card by card basis. I.E. can I assign a single port from a card or do I have to assign the whole card.
Currently each server will have a pair of VIOS and a NIM server as standard, well that's the plan.
it's now sadly some time ago I actively worked with these beside some courses, but here we go (I hope the other guys correct me if there is an error in my explanation):
Basically each port will show up as single interface, like a 4 port NIC assigned to a VIOS will show as ent0, ent1, ent2, ent3.
Today you usually want to bind 2 physical adapters together with a main and a backup adapter which ist called an Etherchannel. You can also bundle adapters together (Link Aggregation) for performance reasons, but you can only have 1 backup adapter and that should be, as said, another physical adapter.
So it depends a lot on your layout how many networks you want to have connected to your systems.
Many setups I had to do with looked like this:
You will also want a HASEA (High Available Shared Ethernet Adapter), so that in case of failure of the whole EtherChannel, the 2nd VIOS can take over - but I leave this out for now.
The EtherChannel is built with smitty etherchannel on the VIOS and don't forget to have no IP-address on the adapters you put together. The IP-address will be assigned later on the new generated SEA!
On the HMC you create a virtual Ethernet Adapter with a , that will be the virtual Ethernet Adapter with a Port VLAN ID (PVID, not the Physical Volume Identifier ) and check the box that it will be the adapter with contact to the outside physical LAN.
You also create the other virtual Ethernet Adapters that will be assigned to each LPAR via HMC, but they don't get the check in the box as they don't have a connection to the outside.
The SEA will be created on the VIOS command line by "fusing" the virtual Ethernet Adapter with the checked box for the outside communication together with the EtherChannel Adapter you created formerly out of the 2 physical adapters.
The assignment, which virtual Adapter in an LPAR corresponds to which SEA is solely defined by the PVID they share, in the "painting" above I took PVID 11 as example.
Next to this setup for the ADMIN LAN, you will probably have such a construct for a PROD LAN and a BACKUP LAN (network backup), but I was too lazy to "paint" them next to this
With 4x4 ports per box, you have different options now. If you have a very big load on the NICs, you can either bundle ent0, ent1, ent2, ent3 together in a Link Aggregation, so that they all 4 form up the "main" adapter in the EtherChannel and you get a lot of possible throughput. Though you will have to take a port of another physical adapter as the backup adapter for your Etherchannel. Though what to do with the rest of 3 ports on this physical adapter..?
If you don't need for aggregated ports, then you can setup maybe something like:
Maybe someone else has a better idea how to assign your many ports, but it all depends on what you want/have to achieve.
For SAN, you will have to decide if you go for vSCSI or NPIV.
vSCSI will let you provide LVs, VGs or PVs from the VIOS to the LPARs. The LPAR will see them as normal hdisks.
NPIV is N-Port Virtualization and will virtualize a FC-Adapter which then can be assigned to the differnt LPARs. Doing this will create 2 new WWNs, 1 of these is for LPM (Live Partition Mobility), to which the LUNs from your storage will be zoned as usual.
I personally like NPIV more because the administration of the mappings for LUN/DISK-to-LPAR that comes with vSCSI is much more tideous work to handle instead of the zonings and the NPIV-WWNs assignments to the LPARs. The downside is, that you have to do driver updates on each LPAR where vSCSI will be updated on the involved VIOS.
This as a start and quick overview for your considerations.
The HASEAs, which you want to have for sure too, needs some more details when configuring the SEA, but I assume you will have to make your layout 1st.
Last edited by zaxxon; 08-12-2016 at 03:57 AM..
Reason: additions, corrections and more corrections
These 2 Users Gave Thanks to zaxxon For This Post:
Thanks very much for the info, I had read a sizable chunk of the VIOS config and implementation manual and had worked some of the above out, but there's some really handy tips in there. For the SAN connectivity it will be NPIV through the SVC for onward attachment of the LUNs.
The network configs will be pretty complex here as there are numerous VLAN's, at the moment I have just started the Low Level Design - so just a couple of pictures at the moment (one attached).
you're absolutely welcome. It also makes me recapitulate these very interessting things again
I don't want to be a nitpicker, but each VIOS is his own OS/LPAR. VIOS #2 will start counting devices from ent0 again.
You wrote you have 1 HMC - quite a bit dangerous if it fails. The LPARs and VIOS will not complain and run if the HMC is down, but you can't do anything on the Managed Systems for this time.
Also remember that each HMC has a dhcpd which provides internal IP-addresses for the service processors (SP). If you have more than one, they may not see each other. They are attached to a NIC per Managed System often very simple by a plain network hub.
I do not completely understand the layout, tbh.
Does the same colors mean, that they are in the same EtherChannel? For instance the yellow
VIOS #1: ent0, ent1
VIOS #2: ent16
Which of the 3 is the main, which the backup? Are 2 of them supposed to form up as an aggregation as main adapter so you have a higher bandwith?
In terms of hardware failure, it would not make sense to use 1 port as main and 1 as backup when they are on the same adapter.
This might work if only 1 port fails, but usually the whole adapter says goodbye and then there would be nothing left.
Not sure if intended but you cannot form an EtherChannel from adapters of VIOS #1 and VIOS #2. You can only create an EtherChannel with adapters from the same OS/LPAR/VIOS.
Later on the upper layers you can hand over traffic to a vNIC on another VIOS with the HASEA.
For clarity - an EtherChannel is good for 2 things:
To have a backup NIC available when one of the physical NICs fail.
To be able to get a higher throughput by aggregating physical adapters as "main" adapter in an EtherChannel. The backup adapter can always be only 1 physical NIC.
If the whole EtherChannel fails, ie. the main and backup physical adapters in it, or the whole VIOS goes down - then the HASEA comes into action. It hands over the traffic to VIOS #2, that is hopefully still up and running
If you describe the requirements on hardware failure or possible aggregation of adapters to get a higher throughput and how many networks are needed, we can maybe assist designing.
You wrote you have 4 x 4port NICs per Managed System - the 2 other are not used on the plan yet - maybe we can do something to make it more failsafe with these.
Hi Guys,
I.E. can I assign a single port from a card or do I have to assign the whole card.
Gull04
As far as i know you have to assign the whole card, except you using the new SRIOV feature.
To assist, i added our current Power8 Dual-VIO-Setup. Maybe this help your planing your infrastructure.
Regards
PS
Assigning a IP directly to an SEA results in crashes an misbehavior of the VIO-Server in our Power8 (S824) environment. On Power7 there are not such problems. So we added dedicated (vlan-tagged) virtual NICs to assign the VIO-Server-IPs.
- this is one big LACP-Channel/SEA for all VLANs
- this SEA must have an PVID which is not used in the real Network and will never called directly
- maybe this link (sea with load sharing) will help
Hi,
I have two physical interface connected to solaris box.
1. e1000g1
2. e1000g2
I have added these interfaced under a same IPMP group "IPMP1"
After that I have configured a test address for e1000g1 like below
ifconfig e1000g1 addif <ip-address> netmask + broadcast + -failover... (1 Reply)
Hi,
$ ioslevel
2.2.4.10
I always get these errors when I perform any action using the web interface of IVM / VIOS
Shutdown Partitions Help
You have chosen to shutdown the following partitions. The recommended shutdown method is to use the client operating systems shutdown... (1 Reply)
Hello,
Lets say for simplicity that I do not use any vlan config inside my server - one lpar group use hea physical port1, another group hea physical port2. Physical port1 configured as vlan1 on external switch, physical port2 as vlan2.
What is the common practice - should I isolate my vios... (0 Replies)
Solaris 10 on an X86 box - Config runs fine has a couple of items in log (really long log file).
When I try make I get
make: Fatal error in reader: Make.inc, line 84: Unexpected end of line seen
line 84 is
LIBRADIUS_WITH_OPENSSL = 1
any ideas on what to try? (1 Reply)
Hi ,
when i configure my server with a fixed ip address in dhcpd.conf as below
subnet 172.21.151.0 netmask 255.255.255.0 {
range 172.21.151.66 172.21.151.66;
}
host switch {
hardware ethernet 00:05:30:02:DB:31;
fixed-address 172.21.151.66;
}
when i start dhcpd server , it... (2 Replies)
Can anybody provide me with usefull links to get knowledge how VIOS works and how to configure Lpars on it? I am tired and feel lazy to search through IBM redbooks :-) .. so pls help me :p (3 Replies)
In sshd_config it is written:
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options change a
# default value.
So does that mean for the following:
... (2 Replies)
When I telnet (ssh) over to my Fedora system, I find the colors horrible. For instance, regular files are white text, which is fine, but directories show up as dark blue which is virtually invisible against the black background). Additionally, when using vi, I find the colors great doing perl... (3 Replies)
Hi -
I am fairly new with unix. I have a unix box, installed with Solaris 2.8. I can get the box hooked up to the internet, not using DHCP but static IP address behind a Linksys firewall. However, I cannot telnet to the unix box or from within the unix box. If I telnet from a windows... (8 Replies)