VM Network Packet Drops with Broadcom bnxtnet Adapter Firmware after upgrade from ESXi 8.0.3 to 9.1.1

Products

• VMware vSphere ESXi

Issue/Introduction

Virtual machines residing on the same VLAN/subnet across different ESXi hosts experience east-west network communication drops following an upgrade to VCF 9.1.1 on Dell hardware.

Symptoms observed:

• Network communication between source VM and destination VM drops after the ICMP request leaves the source ESXi host.

• Packet capture analysis indicates ICMP echo requests successfully leave the source host via the physical NIC uplink.

• ARP requests reach the destination host and VM, and the destination VM replies.

• ARP replies fail to return to the source host across the physical network uplink.

• Upstream top-of-rack switches confirm proper MAC learning across VPC peer links for source and destination MAC addresses, isolating drops to the physical NIC layer.

Environment

 

• VMware Cloud Foundation (VCF) 9.1.1

• The behavior is seen in VMware ESXi 8.0.3  and 9.1.1

• Dell PowerEdge Servers

• Broadcom bnxtnet driver version 238.1.147.0

• Broadcom bnxtnet firmware package versions 38.11.70.00 and 38.11.91.00

 

Cause

A unknown defect in Broadcom bnxtnet network adapter firmware package version 38.11.70.00 (and 38.11.91.00) causes intermittent ARP and return packet drops on physical NIC interfaces under VCF 9.1.1.

Resolution

Engage your hardware vendor for their recommended guidance

Additional Information

We have seen that with a downgrade of the firmware package, the traffic behavior was resolved