VIOS Configuration Question


 
Thread Tools Search this Thread
Operating Systems AIX VIOS Configuration Question
# 1  
Old 08-11-2016
VIOS Configuration Question

Hi Guys,

This is a general question relating to the building of VIO servers;

It has been a long time since I have done any serious AIX work, AIX 4.3 was new and I was for the most part working with 3.2.2 on older RS6000 kit.

So to the current day, I'm about to take delivery of some new tin which I have to plan, install and configure the first batch of tin will comprise.
  • Three Power 8 S824's
  • One HMC
  • One SVC
  • One EMC VNX5800 Disk Array

The S824's are customer install and will each come with 4 x Dual Port 16Gb HBA's and 4 x Quad Port 10Gb NIC's along with the 4 x 1Gb copper NIC.

So in simple terms my question is:- When building the VIO's can I assign LAN and SAN resource on a port by port basis, or do I have to assign a it on a card by card basis. I.E. can I assign a single port from a card or do I have to assign the whole card.

Currently each server will have a pair of VIOS and a NIM server as standard, well that's the plan.

Regards

Gull04
# 2  
Old 08-11-2016
Hi gull04,

it's now sadly some time ago I actively worked with these beside some courses, but here we go (I hope the other guys correct me if there is an error in my explanation):

Basically each port will show up as single interface, like a 4 port NIC assigned to a VIOS will show as ent0, ent1, ent2, ent3.

Today you usually want to bind 2 physical adapters together with a main and a backup adapter which ist called an Etherchannel. You can also bundle adapters together (Link Aggregation) for performance reasons, but you can only have 1 backup adapter and that should be, as said, another physical adapter.
So it depends a lot on your layout how many networks you want to have connected to your systems.

Many setups I had to do with looked like this:

Code:
          ADMIN-LAN

        SWITCH    SWITCH
           |        |
           |        |
           |        |
           |        |
          ent0     ent1
           | main   | backup
           +--------+
                |
                |
               ent 2
                | EtherChannel
                +
                |
               ent 3 bridged vNIC (VLAN11)
                |
                =
                |
               ent 4
                | Shared Ethernet Adapter (SEA)
                | PortVLAN ID 11
                |
                |
                |
                |
                +------------------+-------------------+
                |                  |                   |
                |                  |                   |
                |                  |                   |
               LPAR A             LPAR B              LPAR C
               ent0               ent0                ent0
                  virtual NIC        virtual NIC         virtual NIC
                  PVID 11            PVID 11             PVID 11

You will also want a HASEA (High Available Shared Ethernet Adapter), so that in case of failure of the whole EtherChannel, the 2nd VIOS can take over - but I leave this out for now.

The EtherChannel is built with smitty etherchannel on the VIOS and don't forget to have no IP-address on the adapters you put together. The IP-address will be assigned later on the new generated SEA!

On the HMC you create a virtual Ethernet Adapter with a , that will be the virtual Ethernet Adapter with a Port VLAN ID (PVID, not the Physical Volume Identifier Smilie) and check the box that it will be the adapter with contact to the outside physical LAN.
You also create the other virtual Ethernet Adapters that will be assigned to each LPAR via HMC, but they don't get the check in the box as they don't have a connection to the outside.

The SEA will be created on the VIOS command line by "fusing" the virtual Ethernet Adapter with the checked box for the outside communication together with the EtherChannel Adapter you created formerly out of the 2 physical adapters.

The assignment, which virtual Adapter in an LPAR corresponds to which SEA is solely defined by the PVID they share, in the "painting" above I took PVID 11 as example.

Next to this setup for the ADMIN LAN, you will probably have such a construct for a PROD LAN and a BACKUP LAN (network backup), but I was too lazy to "paint" them next to this Smilie

With 4x4 ports per box, you have different options now. If you have a very big load on the NICs, you can either bundle ent0, ent1, ent2, ent3 together in a Link Aggregation, so that they all 4 form up the "main" adapter in the EtherChannel and you get a lot of possible throughput. Though you will have to take a port of another physical adapter as the backup adapter for your Etherchannel. Though what to do with the rest of 3 ports on this physical adapter..?
If you don't need for aggregated ports, then you can setup maybe something like:

Code:
        physAdapter 1   physAdapter 2   Etherchannel

Port A  ent0            ent4            ent0 + ent4 = ent8
Port B  ent1            ent5            ent1 + ent5 = ent9
Port C  ent2            ent6            ent2 + ent6 = ent10
Port D  ent3            ent7            ent3 + ent7 = ent11
                                         |      |
                                        main + backup

Maybe someone else has a better idea how to assign your many ports, but it all depends on what you want/have to achieve.

For SAN, you will have to decide if you go for vSCSI or NPIV.
vSCSI will let you provide LVs, VGs or PVs from the VIOS to the LPARs. The LPAR will see them as normal hdisks.

NPIV is N-Port Virtualization and will virtualize a FC-Adapter which then can be assigned to the differnt LPARs. Doing this will create 2 new WWNs, 1 of these is for LPM (Live Partition Mobility), to which the LUNs from your storage will be zoned as usual.

I personally like NPIV more because the administration of the mappings for LUN/DISK-to-LPAR that comes with vSCSI is much more tideous work to handle instead of the zonings and the NPIV-WWNs assignments to the LPARs. The downside is, that you have to do driver updates on each LPAR where vSCSI will be updated on the involved VIOS.

This as a start and quick overview for your considerations.
The HASEAs, which you want to have for sure too, needs some more details when configuring the SEA, but I assume you will have to make your layout 1st.

Last edited by zaxxon; 08-12-2016 at 03:57 AM.. Reason: additions, corrections and more corrections
These 2 Users Gave Thanks to zaxxon For This Post:
# 3  
Old 08-12-2016
Hi zaxxon,

Thanks very much for the info, I had read a sizable chunk of the VIOS config and implementation manual and had worked some of the above out, but there's some really handy tips in there. For the SAN connectivity it will be NPIV through the SVC for onward attachment of the LUNs.

The network configs will be pretty complex here as there are numerous VLAN's, at the moment I have just started the Low Level Design - so just a couple of pictures at the moment (one attached).

Anyway, thanks for the most welcome reply.

Regards

Gull04
VIOS Configuration Question-vios-configpng
# 4  
Old 08-12-2016
Hi gull04,

you're absolutely welcome. It also makes me recapitulate these very interessting things again Smilie

I don't want to be a nitpicker, but each VIOS is his own OS/LPAR. VIOS #2 will start counting devices from ent0 again.
You wrote you have 1 HMC - quite a bit dangerous if it fails. The LPARs and VIOS will not complain and run if the HMC is down, but you can't do anything on the Managed Systems for this time.
Also remember that each HMC has a dhcpd which provides internal IP-addresses for the service processors (SP). If you have more than one, they may not see each other. They are attached to a NIC per Managed System often very simple by a plain network hub.

I do not completely understand the layout, tbh.

Does the same colors mean, that they are in the same EtherChannel? For instance the yellow

VIOS #1: ent0, ent1
VIOS #2: ent16

Which of the 3 is the main, which the backup? Are 2 of them supposed to form up as an aggregation as main adapter so you have a higher bandwith?

In terms of hardware failure, it would not make sense to use 1 port as main and 1 as backup when they are on the same adapter.
This might work if only 1 port fails, but usually the whole adapter says goodbye and then there would be nothing left.

Not sure if intended but you cannot form an EtherChannel from adapters of VIOS #1 and VIOS #2. You can only create an EtherChannel with adapters from the same OS/LPAR/VIOS.
Later on the upper layers you can hand over traffic to a vNIC on another VIOS with the HASEA.

For clarity - an EtherChannel is good for 2 things:
  • To have a backup NIC available when one of the physical NICs fail.
  • To be able to get a higher throughput by aggregating physical adapters as "main" adapter in an EtherChannel. The backup adapter can always be only 1 physical NIC.

If the whole EtherChannel fails, ie. the main and backup physical adapters in it, or the whole VIOS goes down - then the HASEA comes into action. It hands over the traffic to VIOS #2, that is hopefully still up and running Smilie

If you describe the requirements on hardware failure or possible aggregation of adapters to get a higher throughput and how many networks are needed, we can maybe assist designing.
You wrote you have 4 x 4port NICs per Managed System - the 2 other are not used on the plan yet - maybe we can do something to make it more failsafe with these.

Cheers
zaxxon

Last edited by zaxxon; 08-12-2016 at 09:45 AM..
# 5  
Old 08-12-2016
Quote:
Originally Posted by gull04
Hi Guys,
I.E. can I assign a single port from a card or do I have to assign the whole card.
Gull04
As far as i know you have to assign the whole card, except you using the new SRIOV feature.

To assist, i added our current Power8 Dual-VIO-Setup. Maybe this help your planing your infrastructure.

Regards

PS
Assigning a IP directly to an SEA results in crashes an misbehavior of the VIO-Server in our Power8 (S824) environment. On Power7 there are not such problems. So we added dedicated (vlan-tagged) virtual NICs to assign the VIO-Server-IPs.

Last edited by -=XrAy=-; 08-12-2016 at 11:12 AM..
This User Gave Thanks to -=XrAy=- For This Post:
# 6  
Old 08-12-2016
Hi XrAy Smilie

Just to make sure I understand the plan you attached:
  • The bundle of EtherChannels and SEAs is just shown abstract I assume?
  • The "abstract?" SEA has a PVID of 2999?
  • If there is no abstract bundle, have the vNICs just tagged more than 1 VLAN ID and all are bundled via LACP to a big one?
# 7  
Old 08-12-2016
Hi zaxxo,

- this is one big LACP-Channel/SEA for all VLANs
- this SEA must have an PVID which is not used in the real Network and will never called directly
- maybe this link (sea with load sharing) will help Smilie

Regards

Moderator's Comments:
Mod Comment Removed www.unix.com/ from the link.

Last edited by zaxxon; 08-12-2016 at 12:41 PM..
These 2 Users Gave Thanks to -=XrAy=- For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Solaris

IPMP Configuration Question/Problem

Hi, I have two physical interface connected to solaris box. 1. e1000g1 2. e1000g2 I have added these interfaced under a same IPMP group "IPMP1" After that I have configured a test address for e1000g1 like below ifconfig e1000g1 addif <ip-address> netmask + broadcast + -failover... (1 Reply)
Discussion started by: praveensharma21
1 Replies

2. AIX

VIOS 2.2.4.10 - Errors

Hi, $ ioslevel 2.2.4.10 I always get these errors when I perform any action using the web interface of IVM / VIOS Shutdown Partitions Help You have chosen to shutdown the following partitions. The recommended shutdown method is to use the client operating systems shutdown... (1 Reply)
Discussion started by: filosophizer
1 Replies

3. AIX

VIOS IP address - separate vlan for vios servers ?

Hello, Lets say for simplicity that I do not use any vlan config inside my server - one lpar group use hea physical port1, another group hea physical port2. Physical port1 configured as vlan1 on external switch, physical port2 as vlan2. What is the common practice - should I isolate my vios... (0 Replies)
Discussion started by: vilius
0 Replies

4. UNIX for Advanced & Expert Users

Free Radius Configuration question

Solaris 10 on an X86 box - Config runs fine has a couple of items in log (really long log file). When I try make I get make: Fatal error in reader: Make.inc, line 84: Unexpected end of line seen line 84 is LIBRADIUS_WITH_OPENSSL = 1 any ideas on what to try? (1 Reply)
Discussion started by: NeedLotsofHelp
1 Replies

5. Debian

Question on the dhcp-server configuration for a fixed IP address.

Hi , when i configure my server with a fixed ip address in dhcpd.conf as below subnet 172.21.151.0 netmask 255.255.255.0 { range 172.21.151.66 172.21.151.66; } host switch { hardware ethernet 00:05:30:02:DB:31; fixed-address 172.21.151.66; } when i start dhcpd server , it... (2 Replies)
Discussion started by: Gopi Krishna P
2 Replies

6. AIX

VIOS

Can anybody provide me with usefull links to get knowledge how VIOS works and how to configure Lpars on it? I am tired and feel lazy to search through IBM redbooks :-) .. so pls help me :p (3 Replies)
Discussion started by: wwwzviadi
3 Replies

7. UNIX for Dummies Questions & Answers

OpenSSH Configuration Question

In sshd_config it is written: # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options change a # default value. So does that mean for the following: ... (2 Replies)
Discussion started by: mojoman
2 Replies

8. Debian

gnome file manager configuration question

Is there a way to configure the gnome file manager to open each directory in the same window instead of opening a new one? (Debian 5.0.0) (5 Replies)
Discussion started by: snorkack59
5 Replies

9. Linux

xterm font colors - configuration question?

When I telnet (ssh) over to my Fedora system, I find the colors horrible. For instance, regular files are white text, which is fine, but directories show up as dark blue which is virtually invisible against the black background). Additionally, when using vi, I find the colors great doing perl... (3 Replies)
Discussion started by: ripley
3 Replies

10. UNIX for Dummies Questions & Answers

Configuration Newbie Question

Hi - I am fairly new with unix. I have a unix box, installed with Solaris 2.8. I can get the box hooked up to the internet, not using DHCP but static IP address behind a Linksys firewall. However, I cannot telnet to the unix box or from within the unix box. If I telnet from a windows... (8 Replies)
Discussion started by: trigeek8888
8 Replies
Login or Register to Ask a Question