Solaris

This should not happen if everything is configured properly.

I checked your initial output more carefully (sorry for that )

What looks wrong to me is that you are using L2 aggregation (aggro0 interface), and you have created from that interface two virtual switches, then you used those interfaces to create ipmp group inside ldom.

I don't think that is supported configuration, looks kinda silly

Since you have aggregated two interfaces (net0 and net1) which must be connected to the same physical switch, there is no need to use IPMP inside LDOM (guest domain, i don't think this is supported configuration at all, possibly why you are having mac collisions) or create multiple virtual switches over one interface (aggr0).

This schematic should be more illuminating :

Primary domain (hypervisor - bare metal)
---> net0 <> net1 [aggr0 L2] ---> primary-vsw50 (on primary, created using aggr0, add vsw command) ---> vnet0 for guest ldom1, ldom2 (add-vnet command)

Only one vnet is enough, since if net0 fails, all you will loose is bandwidth from one interface.

No need to tag the interfaces on the hypervisor os level (aggr5000, dladm create-vnet), since this is done for LDOMS on the vsw/vnet level (PVID,VID).
This should work, but it is a legacy way to implement vlan tagging in LDOMS.

As for bare metal domains (primary,secondary), let me offer a short explanation of domains as i understand it...
For instance, you have sparc t4-2 with two sockets, two 4 port network cards and two 2 port FC card.

You can create two hardware domains - primary and secondary, in which the actual I/O hardware is splited between those two domains (each has one PCI card and one FC card and one CPU socket and memory ).

Now you have a situation that you have one t4-2 sparc which is actually two machines separated on hardware level. So all LDOMS created on primary domain will use its resources (CPU,PCI - half of them) and ldoms on secondary will use its resources (other half)

Basically, if one socket fails due to hardware failure, only the primary domain and guest ldoms on them will fail, while secondary and guest ldoms on it will continue to run.
Those setups complicate things considerably and are done on machines which have more resources in redundant matter (like 4 cards or 4 sockets, 2 phys cards per domain for redundancy etc.)

For your setup i guess you need (keep it simple - as per scheme in the begining) :

One primary domain (bare metal)
One vsw created on top of the aggr0 interface in primary domain.
One vnet interface added to LDOM from primary-vsw on primary domain.
One VDS (virtual disk service) in primary domain per guest ldom (sneezy-vds@primary, otherguestldom-vds@primary etc.) in which you add disks for ldoms.

Hope that clears things out.

Regards
Peasant.

Hi Peasant, again, I cant thank you enough for your input.

So what we actually have is a t5-2 which has two sockets, 2x two port FC Cards and 4x gigabit Ethernet ports.

As you said the machine is split right down the middle with each root complex owning exactly half of the hardware including local Hard drives.

What we have is:
1x Primary Control domain (Control, IO, Service). Obviously all LDOMS are managed from the Primary.

1x Secondary (Or what some people call 'Alternate') IO, Service domain which can see bare metal Storage.


Im sure im telling you what you already know but it help me explain it out The idea of us have two IO, service domains (Priamry and Secondary) is that we can actually take one of them down (i.e for patching) and all Guest LDOMS will continue to run, route traffic in/out, see LUNS etc.

And this is the case. When i init 6 or shutdown the primary LDOM, all Guests continue to operate via the Secndary (Alternate) Domain. And vice-a-versa.



So when I create a guest LDOM, i make sure to create two VNET's, one pointing to the Primary VSW and the other to the Secondary VSW. And when creating new LDOMS, i alterante which switch vnet0 point to so that all traffic does always go through one switch.

And this is the same principle for DISKS, i use multipathing groups (MPGROUP) to ensure that guest can see LUNs from both IO, SERVICE domains.


I think you are correct about the IPMP guest settings, I am just reading up more about that.

I also don't pretend to completely understand the difference between the trunk policies (L2, L3 etc.). I am also doing some more reading on that.


FYI, we also have some T5-2 servers which not only have 2x two port FC cards but also 2x two port ehternet cards in addition to the 4x on board ethernet ports. These serers follow the same principle as the on i use in the original post, but onviously each root complex has 4 ethernet ports each for the trunk.

You do have more then one t5-2 machine ?

If you have only one root complex on each (only primary domain) and take care you don't over commit resources (cpu / mem), you should be able to live migrate a ldom from one machine to another.
All you have to watch is that the names are the same for all the virtual devices and the backend devices are the same (naturally)
I am not sure if you can have 2 root complexes (primary / secondary) and do a live migrate of guest ldom inside the same machine.

That is not a use case at all, but i could be wrong (i have never made such setup).

Use case for root complexes is two have complete hardware separation of, for instance, production and test and it is used on machines with more sockets, more cards etc.

Take this example - sneezy is a production ldom and sloppy is a test ldom.
We are using only primary domain only one root complex logically separated production and test

Both t5-2 have same names and configuration

FC (2 FC 2 PORT cards)

1 port from each FC card is for production usage (zoned on switch, production host group on storage)
1 port from each FC card is for test usage (zoned on switch, test host group on storage)

This way if one FC cards dies, production and test will continue to operate.

You just prefix when create a VDS (virtual disk services) with prod or test depending.

So you have sneezy-prodvds and sloppy-testvds on both sparcs in primary domains, with disks added to them according to layout above (test and prod host groups and paths).

Remember you have freedom here to add both to any ldom (test or production disk), only naming policy is telling you to which VDS you will add each disk.

NETWORK (2 LAN 4 PORT cards, 8 ports total, dladm show-phys)

Now you make a choice, will you use aggregation, ipmp or dlmp

Example here is with aggregation (aggr0,aggr1).

You take 2 ports from one card and 2 ports from other card which lead to the same LAN switch and create an aggr0 interface,then create a production-vsw from that interface (with production vlan tags configured pvid/vid)
You add vnet to sneezy ldom from production-vsw (or any other production ldom)

From other network ports (2 from each card remains), you create an aggr1 interface, then create a test-vsw from that interface (with test vlan tags configured on vsw pvid/vid)
You add vnet to sloppy ldom from test-vsw (or any other test ldom).

This way if one card dies, you will loose 2 production and 2 test paths, but both test and production will continue to operate on lower bandwidth (2 x instead of 4 x)

VSW and VDS are named the same on both sparc machines primary domain (this is a requirement for migration (live or cold).

Now you have sneezy production ldom, with production-vsw --> vnet for networking and disk added via proper FC path and added to sneezy-prodvds and sneezy ldom.

Also you have sloppy test ldom, with test-vsw --> vnet for networking and disk added via proper FC path and added to sloppy-testvds and sloppy ldom.

PRIMARY DOMAIN network with tagging on both sparcs :

I recommend using a separate vlan for primary domain IP addressing (control domain can be isolated on vlan layer on network for security reasons)

Since you have tagging on switch and aggr0/1 interfaces, you will have to create a tagged interface for primary domain.
dladm create-vlan -l aggr1 -v <your vlanid> vlan-link
If you don't provide the vlan-link it will create a aggr1<yourvlanid> interface.

This is the interface you will use to create ip address for primary domain on both machines. We are using aggr1 here (test network) for live migration of all ldoms (both production and test), but you are free to choose any (0 or 1) depending on the network topology and bandwidth required.

Now you can issue a test live migration from host1 to host2 with command
ldm migrate -n sloppy host2 # -n switch is just to check if migration will work and it is a great way to check if configuration on both physical machines are the same.

Final result that you can migrate any production or test guest LDOM to any sparc t5-2 machine without of interrupting the service (live migrate) or cold migrate (with ldom down), while having LAN and FC resources seperated for production and test and having more or less "keep it simple" configuration

Be sure firmware levels are the same on all your sparc t5-2 machines for live migrate to work (cold migrate will not have this limit).

Hope that helps.

Regards
Peasant.