Network stack failure and kernel deadlock triggered by VMXNET3 port lock contention on ESXi Hosts.

Products

• VMware vSphere ESXi

Issue/Introduction

• Guest VM CPU usage drops to 0MHz or significantly below normal idle thresholds.
• vCenter Server reports the ESXi host as "Not Responding."
• Direct console access to the ESXi host remains unresponsive to keyboard interrupts (F2).
  
  In the vmkernel.log, the following error signatures are present:
  
  NetPort: 708: Failed to acquire port non-exclusive lock 0x####[Failure]
  
  PCIVPD: VPD capability is not found
  
  lm_hwrm_nvm_get_variable failed

Environment

VMware vSphere ESXi

Cause

The issue is caused by a kernel race condition involving the virtual switch (vDS) and the transmit (TX) datapath. 
When high-throughput operations occur—specifically triggered by datastore rebalancing (Storage DRS) or the enabling of high-resource VM NICs—the kernel fails to acquire necessary non-exclusive locks for the VMXNET3 ports. 
This leads to a kernel deadlock, crashing the network stack and severing storage connectivity, which results in an All Paths Down (APD) state that stuns the guest VMs.

Resolution

To resolve this issue, you might want to  align the physical network adapter firmware and the ESXi driver version with the validated combinations on the Hardware Compatibility List (HCL).

Firmware and Driver Alignment:

Identify the physical NICs used for vDS uplinks by running: esxcli network nic list
Update to the latest Dell custom async driver and matching firmware release.

Storage DRS Mitigation :

Change the Storage DRS (SDRS) automation level for the impacted datastore cluster to Manual. 
This prevents automated storage rebalancing from triggering the lock contention until patches are applied.