vCenter Network Performance Graphs Show Anomalous Throughput on ESXi 8.0u3

Products

• VMware vCenter Server

Issue/Introduction

• vCenter performance graphs show extreme or unrealistic network throughput/usage at the node level.
• The issue is primarily observed in weekly and monthly historical views, while real-time graphs may appear normal.
• Physical switch telemetry does not reflect the traffic volume shown in vCenter.
• Affected hosts are running ESXi 8.0u3 with bnxtnet driver version 236.1.128.0-1OEM.

vmkernel.log:

XXXX-XX-XXTXX:XX:XX.XXXZ In(182) vmkernel: cpu36:2320994)bnxtnet: dev_dump_fw_trace:159: [vmnic5 : 0x45215eb40000] Dumping FW Trace:
XXXX-XX-XXTXX:XX:XX.XXXZ In(182) vmkernel: cpu36:2320994)bnxtnet: dev_print_fw_data:126: [vmnic5 : 0x45215eb40000] == START OF TRACE ==
XXXX-XX-XXTXX:XX:XX.XXXZIn(182) vmkernel: cpu36:2320994)bnxtnet: dev_print_fw_data:138: [vmnic5 : 0x45215eb40000] == END OF TRACE ==

XXXX-XX-XXTXX:XX:XX.XXXZ In(182) vmkernel: [00697466] <wrn> health_check: CRT HW SOFT ERROR: Code=0x11F4 Reg=te_nic_int_sts_1 Addr=0x8243001C d1=0x0 d2=0x0\
 

Environment

• vSphere vCenter 8.0 U3
• ESXi 8.0 U3

Cause

The issue is caused by a telemetry calculation anomaly in the bnxtnet driver version 236.1.128.0-1OEM. Under certain conditions, hardware queue stats reset or overflow incorrectly during device quiesce/activate operations, leading the driver to report erroneous "Bytes Received" counters to vCenter.

Resolution

To resolve this issue, align the driver and firmware with the VMware Compatibility Guide (HCL) and update the driver to a below version.

• The older driver (233.0.156.0) resolves the issue.
• https://compatibilityguide.broadcom.com/detail?program=io&productId=50746
• Dell engineering is already aware of this issue and working with Broadcom internally. Until then use the driver 233.0.156.0