VM's connected to a vSphere Standard Switch (VSS) lose network connectivity when they are moved to a vSphere Distributed Switch (VDS)

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

VM's connected to a standard switch via a standard port group tagged with a specific VLAN are unable to communicate to VM's/IP addresses located outside of the ESXi host where they reside after being moved to a distributed port group (connected to a distributed switch) tagged with the same specific VLAN .

• You have confirmed that the TOR switchport connected to the distributed switch via one or more vmnics is configured exactly the same as the TOR switchport connected to the vmnics(s) used by the standard switch.
• You may have moved the vmnic that had been used when the VM connected through a standard switch over to the distributed switch and found that you still see the same behavior. 

Environment

VMware vSphere ESXi
VMware vCenter Server

Cause

• When connected through a VDS:
  • When you tag a Port Group with a VLAN ID (e.g., VLAN 20), the ESXi host is responsible for adding the 802.1Q tag to all outgoing (egress) frames. This is known as Virtual Switch Tagging (VST).
  • If the physical Top-of-Rack (ToR) switchport is configured as an Access Port (or not enabled for trunking), it expects untagged frames.
  • When the ESXi host (via the VDS) sends a frame tagged with VLAN 20 to an Access Port, the physical switch sees the 802.1Q header as an invalid frame format or a VLAN mismatch and drops the traffic.
    
    
• When connected through a VSS: 
  
  • Usually, this traffic would also get dropped due to the physical switch port being configured as an Access port and not expecting to see the 802.1Q tag.  
  • An anomaly with how some physical NIC drivers interact with standard switches (VSS) to handle traffic means that when connected to a standard switch, this traffic is not being dropped as expected. 

Summary:

• Some physical NIC drivers, when bound to a VSS, might strip the VLAN tag before it hits the wire if the physical switchport isn't negotiating a trunk, effectively "flattening" the traffic. The VDS, with its more advanced architecture, often bypasses this specific driver behavior and enforces the tagging at a higher level, sending the tag to the wire regardless.

Resolution

Apply the following configuration guidelines (using standard Cisco IOS syntax as an example):

• Force Trunk Mode: Statically define the port as a trunk to allow tagged frames:
  > switchport mode trunk
  
  
• Define Allowed VLANs (Optional but Recommended). Restrict the trunk to only the VLANs required by the vSphere environment to limit the broadcast domain scope:
  > switchport trunk allowed vlan <VLAN_IDs>