Packet drops on vMotion VMkernel ports due to Intel i40en driver and firmware incompatibility

Products

• VMware vSphere ESXi

Issue/Introduction

When performing network connectivity tests (such as a ping) between vMotion VMkernel ports from an ESXi host SSH session, you may observe intermittent packet loss.

 

• Ping drops occurring between two vMotion VMkernel ports.

• A corresponding increase in receive (RX) packet drops within the physical NIC statistics (vmnic stats) during the period of packet loss.

• These drops occur at the hardware level, suggesting the NIC is discarding packets received from the physical switch port.

 

Environment

VMware vSphere ESXi 8.0.x

Cause

The issue is caused by a compatibility mismatch between the installed network driver and the NIC firmware. According to the VMware Compatibility Guide (VCG), the i40en driver version 2.9.2.0 requires a minimum firmware version of 9.50. Running this driver on older firmware versions (such as 9.10) results in unsupported behavior and hardware-level packet drops.

Resolution

The issue is caused by a compatibility mismatch between the installed network driver and the NIC firmware. According to the Broadcom Compatibility Guide, the i40en driver version 2.9.2.0 requires a minimum firmware version of 9.50. Running this driver on older firmware versions (such as 9.10) results in unsupported behavior and hardware-level packet drops.

Resolution

To resolve this issue, you must upgrade the network card firmware to a version that is compatible with your driver as specified in the Broadcom Compatibility Guide.

1. Verify your current driver and firmware versions by running the following command on the ESXi host:

   esxcli network nic get -n vmnicX (Replace vmnicX with your actual vMotion uplink).

2. Consult the VMware Compatibility Guide to identify the required firmware for your specific driver version.

3. Contact your hardware vendor to obtain the appropriate firmware update package (version 9.50 or higher for driver 2.9.2.0).

4. Perform the firmware upgrade following the vendor's recommended procedures.

5. After the upgrade, reboot the ESXi host and verify that packet drops are no longer occurring during ping tests.

Additional Information

Always ensure that the driver and firmware combination is listed as a supported pair on the Broadcom Compatibility Guide to maintain a stable environment.