The target was simple, we have an internal cloud datacenter at work that provides users and customers both virtual machines and physical machines. Each machine has to network interface cards (NICs), one is in control of the user/customer using SDN layer, the secondary NIC is for our support to monitor and help troubleshoot these machines when needed. This second NIC will be our target today. In the past we used per-user or per-customer firewall separations that was configuration intensive nightmare, but was reliable. However since we learned private VLANs are now supported by vmWares Distributed vSwitch (dvSwitch), we immediately tried to make it cooperate with private VLANs on physical switches. And since it worked like a charm, let me share with you a quick lab example. But theory first!
Theory of separating management rail between different customers with and without private VLANs
Fortunately Private VLANs arrived for most major vendors and promissed the ability to have one giant subnet and separate every host from each other on L2 using some basic principle of declaring ports as either promiscuous (can talk to any other port type), community (can talk to ports of the same community or promiscuous) and isolated (can really talk to only the promiscuous ports). For our need, we can simply declare the firewall port to be promiscous (so it can accept all traffic) and all other ports as isolated (so they can only talk to the firewall and not each other) and this would be a clear case of private VLANs. HOWEVER if only the network was full of physical servers only on any network hardware vendor, be it Cisco, Juniper or HPN. But what happens when virtual machines enter this scenario? Well, because hypervisors use internal software switches, the VMs might leak communication to each other since private vlans are only enforcing separation on physical links. Look at this scenario with VM behind hypervisor:
So what can we do here to solve this between VMs and PMs ? Simple, teach the virtual switches the private VLANs principle of using dot1q tags on trunks. You see, the private VLAN standard declared a way how to do private vlan in multi-switch topology by using primary (for promiscous ports) and secondary VLAN tags (community/isolated) and marking packets based on these tags based on source port type whenever the switch needed to push the packet via a trunk. There is a nice explanation on packetlife.net here of this, so I will jump right at applying this to virtual switches.
First, lets create in our lab two VLAN tag IDs, I will use VLAN 1000 as primary and VLAN 1001 as secondary.
This means we can create a single shared subnet for management of all customer systems, both VMs and PMs, but we need to configure private VLANs on all physical switches, all virtual switches and use the same VLAN tages for primary and secondary VLANs. Our target here will be something like this:
Now let’s get practical, this was my lab setup below on picture. Target is to achieve complete isolation of all VMs and PMs on the network, while at the same time allow all these systems to use default gateway, which was a VRRP IP on interface vlan on the switches.
I was using ESX hosts version 6.0 and HPN 5940 series switches with Comware 7 – R2508 version.
Configuring HPN switches for Private VLANs
Ok, lets configure the HPN’s 5940 first, it all starts with the private VLAN definition:
#Create primary privte vlan vlan 1000 private-vlan primary name Management Private VLAN quit #Create secondary vlan and define it as "isolated" type vlan 1001 # THESE TWO COMMANDS BECAUSE VLAN 1001 can talk to VLAN 1001 private-vlan isolated undo private-vlan community quit #Go back to primary vlan and tie it with secondary VLAN vlan 1000 private-vlan secondary 1001 quit
Next lets look at the interfaces towards ESX hosts, this is easy, we simply define them as normal trunks! So if you already had these as trunks, there is nothing much else to do here and the TenGigabit 1/0/1 interface in my lab looks like this:
interface Ten-GigabitEthernet1/0/1 port link-mode bridge description link to ESX Nodes port link-type trunk port trunk permit vlan all
Following up with port towards the host, and this is interesting, because you need to configure your port like this:
interface Ten-GigabitEthernet1/0/2 port link-mode bridge description link to physical server to isolate port link-type access port access vlan 1001 port private-vlan host
But the funny part is, that the private-vlan host command at the end is actually a macro, that will extend itself into several hybrid
interface Ten-GigabitEthernet1/0/2 port link-mode bridge port link-type hybrid undo port hybrid vlan 1 port hybrid vlan 1000 to 1001 untagged port hybrid pvid vlan 1001 port private-vlan host
And the grand finale, creating a promiscuous gateway interface is usually done on a physical port towards a router, but in my scenario (and actually our production scenarios) we need to create a gateway router as vlan interface locally on a nearby switch, luckily, HPN switches support this and the configuration look like this:
# Switch 1 interface Vlan-interface1000 private-vlan secondary 1001 ip address 126.96.36.199 255.255.252.0 vrrp vrid 1 virtual-ip 188.8.131.52 vrrp vrid 1 priority 110 # Switch 2 interface Vlan-interface1000 private-vlan secondary 1001 ip address 184.108.40.206 255.255.252.0 vrrp vrid 1 virtual-ip 220.127.116.11 vrrp vrid 1 priority 90
Well, that is, only a few extra commands on top of the very familiar vlan declarations and interface configurations, so this should not be any issue for you. And I would guess also for other vendors.
Configuring vCenter’s Distribuged vSwitch (dvSwitch) for Private VLANs
Moving on to the vmWare world, this part was actually what I didn’t know originally and needed to research (link below in external guides links). But it is very easy indeed as this is really just a few configuration steps on a centralized dvSwitch settings.
Note/Disclaimer: This guide expects that you have an ESX host that is using a Distributed vSwitch (dvSwitch) deployed across all your ESX nodes. In my lab I had two nodes called Node #1 and Node #2 (as visible in the lab topology picture above).
Step #1: Find the dvSwitch entity and configure the VLAN 1000 as primary and 1001 as secondary – isolated.
Step #2: In the configuration of the portgroup, where your VMs are connected to, assigned it to the isolated VLAN.
Results & Summary
As was our target all along, after this configuration, all the VMs and the physical server in the topology are not capable of accessing each other as they are members of the isolated secondary VLAN 1001, but they can access the promiscuous gateway interface (hosting VRRP gateway IP) because it is assigned to primary VLAN 1000.
See external guides:
- ESX – Private VLANs Configuration Knowledge Database
- HP Networking Private VLANs configuration PDF (for 5940 series switch)
- Private VLANs basic explanation – courtesy of Packet Life