Intermittent packet loss may occur when bridging is configured on NSX or using HCX Network Extension

Products

• VMware NSX
• VMware vSphere ESXi
• VMware NSX-T Data Center
• VMware HCX

Issue/Introduction

Intermittent packet loss may occur when you've configured bridging (say for L2VPN network extension or HCX network extension) when the following conditions are met:

• NIC driver with VMDQ loopback feature in Intel NIC is used(e.g, Intel X710, E810, XXV710)
• NIC driver for Broadcom network cards (e.g., NetXtreme, BCM57412)
• MAC learning feature on vDS is enabled on NSX Edge Node that is performing bridging or HCX network extension is being used. (Or Edge VM can be connected to NSX segment and have MAC learning is enabled)

Environment

VMware vSphere ESXi 8.0
VMware vSphere ESXi 7.0
VMware NSX-T Data Center
VMware NSX
VMware HCX

Cause

VMDQ loopback feature used by Intel NIC can cause connectivity issues when MAC learning is used on NSX Edge Node (with bridging) or HCX network extension is being used. The broadcast packets reflected by VMDQ loopback feature on physical NIC can cause MACs learned from edge node to move to uplink and as a result some packets may get lost.

Resolution

To turn OFF VMDQ loopback feature on NICs of Hosts where workloads are present as well as Hosts where bridge Edge VMs are present.

 

To turn off VMDQ loopback feature in Intel NIC (on an ESXi 7.0 host) follow the steps below:

1. Install a Intel esxcli plug-in tool by following the Intel esxcli plug-in for managing Intel(r) Ethernet Network Adapters (66772). 

2. Run the following command in SSH console.

   # esxcli intnet misc vmdqlb -e 0 -n vmnicX

Note: The above configuration to disable VMDQ loopback feature is not consistent across reboot. To make this setting persistent, please add the command to rc.local by following Modifying the rc.local or local.sh file in ESX/ESXi to execute commands while booting (2043564)

 

Note: For the Intel E810 NIC with the icen driver, the VMDQ loopback feature is only available in version 1.14 and later. Therefore, for this NIC, the only solution is to upgrade the driver to version 1.14 or higher.

 

 

To turn off VMDQ loopback feature on the unified i40en VMware ESX Driver for Intel(R) Ethernet Controllers X710, XL710, XXV710, and X722 family (for a NIC that is on an ESXi 8.0 host):

 

 1. Update the Intel NIC firmware driver to 2.9.2.0

 2. Disable VMDQ on all vmnicX:

   # esxcli intnet misc vmdqlb set -l 0 -n vmnicX

---

Note:

VMDQ loopback feature is disabled by default with i40en 2.9.2 or later and icen 1.14.2 or later.
Refer to the release notes of the drivers for more details.

Note2:

Inbox driver does not have a feature to disable VMDQ loopback.

 

For Broadcom network cards:
This issue has been observed with driver version 229.0.146.0 and firmware 223.0.205.0 / pkg 22.31.13.70, but not with the latest driver version 232.0.254.0 and its corresponding firmware. For more information on how to download and install the driver, please refer to the KB article: Download and install async drivers in VMware ESXi.  

Additional Information

VMDQ loopback feature is used to allow the communication between a VF and a VM/vmknic which locate on the same pNIC. There is no harm disable this feature if SRIOV is not in used.