Back for another article about LDoms - today we'll cover some tricky networking options that come up if you want to run Solaris 11 zones in LDom guest systems. So what's the problem?
Let's look at what happens with MAC addresses when you create a guest system with a single vnet network device. By default, the LDoms Manager selects a MAC address for the new vnet device. This MAC address is managed in the vswitch, and ethernet packets from and to that MAC address can flow between the vnet device, the vswitch and the outside world. The ethernet switch on the outside will learn about this new MAC address, too. Of course, if you assign a MAC address manually, this works the same way. This situation is shown in the diagram at the right. The important thing to note here is that the vnet device in the guest system will have exactly one MAC address, and no "spare slots" with additional addresses.
Add zones into the picture. With Solaris 10, the situation is simple. The default behaviour will be a "shared IP" zone, where traffic from the non-global zone will use the IP (and thus ethernet) stack from the global zone. No additional MAC addresses required. Since you don't have further "physical" interfaces, there's no temptation to use "exclusive IP" for that zone, except if you'd use a tagged VLAN interface. But again, this wouldn't need another MAC address.
With Solaris 11, this changes fundamentally. Solaris 11, by default, will create a so called "anet" device for any new zone. This device is created using the new Solaris 11 network stack, and is simply a virtual NIC. As such, it will have a MAC address. The default behaviour is to generate a random MAC address. However, this random MAC address will not be known to the vswitch in the IO domain and to the vnet device in the global zone, and starting such a zone will fail.
The solution is to allow the vnet device of the LDoms guest to provide more than one MAC address, similar to typical physical NICs which have support for numerous MAC addresses in "slots" that they manage. This feature has been added to Oracle VM Server for SPARC in version 3.0.0.2. Jeff Savit wrote about it in his blog, showing a nice example of how things fail without this feature, and how they work with it. Of course, the same solution will also work if your global zone uses vnics for purposes other than zones.
To make this work, you need to do two things:
- Configure the vnet device to have more than one MAC address. This is done using the new option "alt-mac-addrs" with either ldm add-vnet or ldm set-vnet. You can either provide manually selected MAC addresses here, or rely on LDoms Manager to use it's MAC address selection algorithm to provide one.
- Configure the zone to use the "auto" option instead of "random" for selecting a MAC address. This will cause the zone to query the NIC for available MAC addresses instead of coming up with one and making the NIC accept it.
I will not go into the details of how this is configured, as this is very nicely covered by Jeff's blog entry already. I do want to add that you might see similar issues with layered virtual networking in other virtualization solutions: Running Solaris 11 vnics or zones with exclusive IP in VirtualBox, OVM x86 or VMware will show the very same behaviour. I don't know if/when these thechnologies will provide a solution similar to what we now have with LDoms.