Packet Loss on VMXNET3 Adapters Due to Driver/Firmware Mismatch.

Products

• VMware vSphere ESXi

Issue/Introduction

Aria Operations and vCenter performance charts may show significant received packet drops (RX) on ESXi hosts.

In this scenario:

• Proactive alerts indicate packet loss on clusters hosting high-load VMs.

• Millions of packet drops are observed at the physical NIC (vmnic) level.

• VM is showing a lot of retransmits. 

• Transitioning from E1000 to VMXNET3 adapters does not resolve the drops.

• Performance monitoring shows high vCPU utilization (frequently reaching 100%) during peak loads.

Environment

VMware ESXi 8.#

Cause

The root cause is an unsupported driver and firmware combination on the physical network adapters. Example, running a newer bnxtnet driver (e.g., 236.#) with older firmware (e.g., 229.#/232.#) leads to buffer management inefficiencies and massive packet drops at the physical layer, which propagates up to the virtual machines.

Resolution

To resolve the packet loss, ensure the physical NIC driver and firmware are aligned according to the VMware Compatibility Guide:

1. Verify Compatibility: Check the VMware Compatibility Guide to confirm the supported firmware version for your specific driver version.

2. Firmware Upgrade: Contact your hardware vendor to upgrade the NIC firmware to the version recommended for the 236.# driver.

3. Validation: After the upgrade, monitor the vmnic stats via CLI or vCenter performance charts to verify that RX/TX drops have stabilized.

4. Secondary Optimization (If drops persist):

   • If CPU saturation remains at 100%, consider increasing the Rx/Tx ring buffer sizes on the physical NICs.

   • Adjust the Guest OS VMXNET3 ring buffer sizes to accommodate burst traffic during high CPU contention.

Additional Information

Always maintain driver/firmware alignment to ensure stability and supportability.