Unrealistically high Rx counters on the pNICs of ESXi hosts

Products

• VMware vCenter Server
• VMware vSphere ESXi 8.0

Issue/Introduction

ESXi hosts utilizing Broadcom bnxtnet drivers may report impossibly high network throughput telemetry in vCenter performance charts or Aria Operations.

• You see high Rx counter values on the pNICs "Bytes received"
• Received average utilization calculated from the total received bytes counter is higher than the actual link speed of the card
• vmnic link speed: 25 Gb/s  (#network nic list or #esxcfg-nics -l)
  
  
  
  NIC statistics for vmnic1: (#esxcli network stats get -n vmnic1)
  
        ...
  
        Bytes received:  22518484621399250 
  
        ....
  
  Calculated: Bytes received: 22518484621399250 / uptime in seconds (1114150) =  20211358094.9 bytes/s = 18.8 GB/s = 150.4 Gb/s
• The "Bytes received" increase over a period of time matches what is observed via "esxtop" network stats "MbRX/s" for the vmnic
• The bnxtnet driver versions lower than 233.x or specifically version 236.1.128.0.
• In some scenarios, the vmnic will report excessive RX packet drops
• This issue often surfaces following an upgrade from ESXi 7 to ESXi 8 if drivers are updated to affected versions like 236.1.173.0.

Environment

• VMware vSphere ESXi 8.0.x
• Broadcom BCM5741x, BCM5750x, or BCM5760x adapters
• bnxtnet driver version 236.1.128.0 or lower

Cause

A telemetry calculation regression in the driver causes hardware queue statistics to reset incorrectly following device quiesce or activation operations. This results in cosmetic counter overflows that do not represent actual packet loss or physical throughput. 
All models BCM5741x, BCM5750x & BCM5760x are affected.

Resolution

Fixed in release bnxtnet driver 238.x and higher. See Download Broadcom products and software for steps to download this release.

To resolve the reporting issue:

1. Identify the current driver and firmware version using esxcli network nic get -n vmnicX.
2. Review the VMware Compatibility Guide to find the certified driver/firmware recipe for your hardware.
3. Upgrade the bnxtnet driver to version 238.x or higher.
4. If an upgrade is not immediately possible, contact your hardware vendor to confirm if reverting to driver version 233.0.156.0 is a supported workaround for your specific firmware level.

 Note: Please validate the NIC card's compatibility before upgrading the driver and firmware.

Additional Information

Troubleshooting NIC errors and other network traffic faults in ESXi

FAQ: Recommendation for drivers/firmware for ESXi hosts

Determining Network/Storage firmware and driver version in ESXi

Download and install async drivers in VMware ESXi

• Verify current driver and firmware versions using 

  esxcli network nic stats get -n vmnic1

• Check for unrealistic Rx bytes by dividing the total received bytes by host uptime in seconds.
• Review the VMware Compatibility Guide for the specific driver/firmware recipe recommended for your hardware model.