Virtual Switch Portgroup configuration for Virtual Machine running containers inside it

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• Virtual Machine requires network connectivity to support multiple nested Docker containers, necessitating the ability to pass multiple 802.1Q VLAN tags through a single virtual network interface (vNIC).
• The following procedure outlines the necessary configurations to support nested container(s) traffic.

Environment

VMware vSphere ESXi

VMware vCenter Server

Resolution

Following configuration changes are required:

• Configure 802.1Q VLAN tagging sub-interfaces directly within the Guest OS network configuration.
• For a vSphere Standard Switch (vSS) portgroup, set the VLAN ID to 4095 to enable Virtual Guest Tagging (VGT).
  • 
• For a vSphere Distributed Switch (vDS) portgroup, configure the VLAN type to VLAN Trunking and specify the required VLAN range.
  • 
• Modify the portgroup Security Policies based on your switch type:
  • For a vSS: Set Promiscuous Mode, MAC Address Changes, and Forged Transmits to Accept.
  • For a vDS (vSphere 6.7+): Enable MAC Learning, and set MAC Address Changes and Forged Transmits to Accept. (Promiscuous Mode can be left as Reject).
• Verify that the upstream physical switch ports connected to the ESXi host uplinks (vmnics) are configured as 802.1Q trunks allowing the specified VLANs.

Additional Information

• To route traffic of multiple containers across different VLANs, the Guest OS must be able to read and process VLAN tags natively. By default, VMware vSphere network configurations utilize Virtual Switch Tagging, where the virtual switch handles 802.1Q tag insertion and removal, delivering only untagged frames to the Guest OS. To change this, the environment must be configured for Virtual Guest Tagging, which passes 802.1Q tagged frames intact directly to the virtual machine.
• Operating in Virtual Guest Tagging mode shifts the responsibility of tag processing from the hypervisor to the Guest OS. Because the ESXi host will now transmit the 802.1Q tagged frames generated by the Guest OS directly out of its physical uplinks, the physical switch ports connected to the ESXi host must be explicitly configured as 802.1Q trunks that allow the required VLAN IDs.Inside the virtual machine, the Guest OS network interface must be configured with 802.1Q sub-interfaces. This setup allows the container runtime to accurately tag outbound traffic and untag inbound traffic assigned to specific container network namespaces.
• Furthermore, the vSwitch security policies must be updated to accommodate the nested containers. By default, the ESXi virtual switch drops any traffic that does not perfectly match the primary MAC address assigned to the virtual machine's network interface. Because individual containers generate their own independent virtual MAC addresses, these default strict security policies will block container traffic. The method used to resolve this drop behavior depends on the type of virtual switch being used.
  • For a Virtual Standard Switch portgroup, setting Promiscuous Mode to Accept overrides the default drop behavior, allowing the interface to receive all incoming Ethernet frames, including those destined for the nested container MAC addresses.
  • For a Virtual Distributed Switch portgroup, enabling MAC Learning is the preferred and highly efficient alternative. MAC Learning dynamically inspects outgoing traffic to learn the unique MAC addresses of the nested containers, ensuring the switch only delivers inbound frames that perfectly match those newly learned addresses.
• Regardless of the switch type, two additional security policies must be modified:
  • Changing MAC Address Changes to Accept permits the Guest OS to dynamically alter the effective MAC address of the interface to match the nested container payloads receiving inbound traffic.
  • setting Forged Transmits to Accept prevents the virtual switch from dropping outbound packets when the source MAC address embedded in the frame differs from the originally registered MAC address.