VM network connectivity fails with 'VlanTag Mismatch' error in pktcap-uw traces.

Products

• VMware vSphere ESXi

Issue/Introduction

Virtual Machines (VMs) on a specific Portgroup are unreachable or experiencing total network isolation. Packet capture analysis using pktcap-uw on the ESXi host reveals the following symptoms:

• Egress ARP requests from the VM are seen leaving the uplink (vmnic) tagged with the correct VLAN ID (e.g., VLAN ##70).

• Ingress ARP replies or return traffic are received on the uplink.

• The ingress traffic is dropped by the vSwitch with the error: Drop Reason 'VlanTag Mismatch'.

• Detailed capture of the ingress traffic (--dir 0) confirms the incoming packets are untagged (VLAN ID 0), despite the Portgroup being configured for a specific VLAN ID.
  
  pktcap-uw --uplink [vmnicX] --dir 0 --mac [Mac_Address_Of_VM]

• The physical switch port is configured with standard trunk port allowing the required VLAN ID (e.g., VLAN ##70).

Environment

VMware vSphere ESXi.

Cause

This issue occurs due to a VLAN tagging mismatch between the virtual switch and the physical switch port configuration.

Specifically, the physical switch port connected to the ESXi host is configured with the VM's target VLAN (e.g., VLAN ##70) set as the Native VLAN.

• IEEE 802.1Q Standard: Traffic belonging to the Native VLAN is transmitted untagged.

• ESXi Behavior: When a Portgroup is configured with a specific VLAN ID (VST Mode), the vSwitch expects incoming traffic to carry that specific 802.1Q tag. If the packet arrives untagged, the vSwitch drops it because it does not match the expected VLAN ID.

Resolution

To resolve this issue, the physical switch configuration must be corrected to ensure the target VLAN is tagged on the trunk link.

1. Identify the VLAN Mismatch Run the following command on the ESXi host to confirm the drop reason:

pktcap-uw --capture Drop --srcip #.#.#.# 

Look for "VlanTag Mismatch" in the drop stats or confirm the absence of 802.1Q headers in the tcpdump output.

2. Correct Physical Switch Configuration Contact the Network Administrator to modify the switch port settings. The target VLAN must be removed from the "Native VLAN" setting though it is added to the "Allowed VLANs" (Tagged) list.

• Example Correction (Cisco IOS):

  • Current (Incorrect): 

    interface GigabitEthernet#/#/#
     switchport mode trunk
     switchport trunk allowed vlan <##70>
     switchport trunk native vlan ##70             <--- Action: Remove this line

3. Verify Connectivity Once the physical switch is reconfigured to tag the traffic:

1. Clear the ARP cache on the guest OS if necessary.

2. Initiate a ping from the VM to the gateway.

3. Verify the pktcap-uw trace now shows incoming packets with the expected 802.1Q tag with expected VLAN (e.g., VLAN ##70).

 

Note: If the packet capture at the physical switch egress confirms that packets are egressing with the correct VLAN tagging, then engage the physical server vendor to address the NIC driver issue. The pktcap-uw capture shows the traffic received by the ESXi kernel from the NIC driver, confirming that the driver is not passing traffic with the correct VLAN tags.

Additional Information

• If the intention is to use the Native VLAN for this VM, the Portgroup on the ESXi host must be configured with VLAN ID 0 (None). However, this is generally not recommended for segmented production traffic.

• Reference:
  Using the pktcap-uw tool in ESXi
  VLAN Tagging issues with Intel X710 network adapters.