Example of private VLAN isolation across Virtual and Physical servers using ESX/dvSwitch and HP Networking Comware switches

The target was simple, we have an internal cloud datacenter at work that provides users and customers both virtual machines and physical machines. Each machine has to network interface cards (NICs), one is in control of the user/customer using SDN layer, the secondary NIC is for our support to monitor and help troubleshoot these machines when needed. This second NIC will be our target today. In the past we used per-user or per-customer firewall separations that was configuration intensive nightmare, but was reliable. However since we learned private VLANs are now supported by vmWares Distributed vSwitch (dvSwitch), we immediately tried to make it cooperate with private VLANs on physical switches. And since it worked like a charm, let me share with you a quick lab example. But theory first!

Theory of separating management rail between different customers with and without private VLANs

Solving management separation with a lot of subnets and firewalls

Fortunately Private VLANs arrived for most major vendors and promissed the ability to have one giant subnet and separate every host from each other on L2 using some basic principle of declaring ports as either promiscuous (can talk to any other port type), community (can talk to ports of the same community or promiscuous) and isolated (can really talk to only the promiscuous ports).  For our need, we can simply declare the firewall port to be promiscous (so it can accept all traffic) and all other ports as isolated (so they can only talk to the firewall and not each other) and this would be a clear case of private VLANs. HOWEVER if only the network was full of physical servers only on any network hardware vendor, be it Cisco, Juniper or HPN. But what happens when virtual machines enter this scenario? Well, because hypervisors use internal software switches, the VMs might leak communication to each other since private vlans are only enforcing separation on physical links. Look at this scenario with VM behind hypervisor:

vmWare’s dvSwitch leaked traffic between two VMs because private VLANs are only implemented on physical switch ports

So what can we do here to solve this between VMs and PMs ? Simple, teach the virtual switches the private VLANs principle of using dot1q tags on trunks. You see, the private VLAN standard declared a way how to do private vlan in multi-switch topology by using primary (for promiscous ports) and secondary VLAN tags (community/isolated) and marking packets based on these tags based on source port type whenever the switch needed to push the packet via a trunk. There is a nice explanation on packetlife.net here of this, so I will jump right at applying this to virtual switches.

First, lets create in our lab two VLAN tag IDs, I will use VLAN 1000 as primary and VLAN 1001 as secondary.

Basic principle of primary and secondary private VLAN IDs when moving between switches

This means we can create a single shared subnet for management of all customer systems, both VMs and PMs, but we need to configure private VLANs on all physical switches, all virtual switches and use the same VLAN tages for primary and secondary VLANs. Our target here will be something like this:

Working private VLAN isolation across VMs and PMs by teaching ESX dvSwitch our private VLAN IDs

LAB Topology

Now let’s get practical, this was my lab setup below on picture. Target is to achieve complete isolation of all VMs and PMs on the network, while at the same time allow all these systems to use default gateway, which was a VRRP IP on interface vlan on the switches.

Private VLANs between VMs and PMs – LAB topology

I was using ESX hosts version 6.0 and HPN 5940 series switches with Comware 7 – R2508 version.

Configuring HPN switches for Private VLANs

Ok, lets configure the HPN’s 5940 first, it all starts with the private VLAN definition:

Next lets look at the interfaces towards ESX hosts, this is easy, we simply define them as normal trunks! So if you already had these as trunks, there is nothing much else to do here and the TenGigabit 1/0/1 interface in my lab looks like this:

Following up with port towards the host, and this is interesting, because you need to configure your port like this:

But the funny part is, that the private-vlan host command at the end is actually a macro, that will extend itself into several hybrid

And the grand finale, creating a promiscuous gateway interface is usually done on a physical port towards a router, but in my scenario (and actually our production scenarios) we need to create a gateway router as vlan interface locally on a nearby switch, luckily, HPN switches support this and the configuration look like this:

Well, that is, only a few extra commands on top of the very familiar vlan declarations and interface configurations, so this should not be any issue for you. And I would guess also for other vendors.

Configuring vCenter’s Distribuged vSwitch (dvSwitch) for Private VLANs

Moving on to the vmWare world, this part was actually what I didn’t know originally and needed to research (link below in external guides links). But it is very easy indeed as this is really just a few configuration steps on a centralized dvSwitch settings.

Note/Disclaimer: This guide expects that you have an ESX host that is using a Distributed vSwitch (dvSwitch) deployed across all your ESX nodes. In my lab I had two nodes called Node #1 and Node #2 (as visible in the lab topology picture above).

Step #1:  Find the dvSwitch entity and configure the VLAN 1000 as primary and 1001 as secondary – isolated.

Find “Private VLAN” settings on distributed switch entity

Configure vlan 1000 as primary and 1001 as secondary with type set to isolated

Step #2: In the configuration of the portgroup, where your VMs are connected to, assigned it to the isolated VLAN.

Find the portgroup in which your VMs are connected to with their interfaces

… assign the portgroup the to the secondary private VLAN 1001

Results & Summary

As was our target all along, after this configuration, all the VMs and the physical server in the topology are not capable of accessing each other as they are members of the isolated secondary VLAN 1001, but they can access the promiscuous gateway interface (hosting VRRP gateway IP) because it is assigned to primary VLAN 1000.

See external guides:


If you enjoyed this blog, please share.

About Peter Havrila

Author's Profile