Unresponsive management console and host lockup on VMware vSphere ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

• The ESXi host locks up and stops logging, showing as 'Not Responding' in vCenter around the reported timestamp.
• There is a logging gap in /var/run/log/vmkernel.log during the time the host is unresponsive.
• The inability to respond to the F2 key on the Out-of-Band Remote Console is the clearest indicator that the entire ESXi kernel is frozen and not processing interrupts.
• A PSOD may or may not appear.
• Linux guest virtual machines reported a vfs: can't find ext4 filesystem error, causing the filesystem to remount as read-only.
• The virtual machines report disk I/O error.
• Shortly before the crash, the /var/run/log/vmkernel.log file contained several of the following entries:

YYYY-MM-DDTHH:MM:SS.###Z In(##) vmkernel: cpu##:#######)NetPort: ###: Failed to acquire port non-exclusive lock 0x400000f[Failure].
YYYY-MM-DDTHH:MM:SS.###Admission failure in path: host/user/vm.####:vmmanon.####
YYYY-MM-DDTHH:MM:SS.###Z In(##) vmkernel: cpu#:#######)NetPort: ###: Failed to acquire port non-exclusive lock 0x4000012[Failure].
YYYY-MM-DDTHH:MM:SS.###Z In(##) vmkernel: cpu##:#######)NetPort: ###: Failed to acquire port non-exclusive lock 0x400000f[Failure].
 
Additional symptoms reported:

• The ESXi host stalls, triggering an HA event in the vSphere cluster and moving all virtual machines to other hosts.
• The host appears to lock up, but it does not result in a purple screen. Once the host is power-cycled, it starts again.

Environment

VMware vSphere ESXi 8.x
VMware vSphere ESXi 7.x

Cause

• A buggy driver or an incompatible NIC driver/firmware causes the ESXi Kernel Deadlock.
• The specific error, Net Port: 708: Fails to acquire port non-exclusive lock 0x40000##[Failure], shows that processes fail to acquire the kernel locks required to manage networking resources like virtual switch ports. This issue escalates under high stress and acts as a known precursor to an ESXi host lockup or PSOD, which indicates a severe race condition or a driver holding the lock indefinitely.

Resolution

1. If the issue recurs, verify that the logs under /var/run/log are updating correctly by running the following command: ls -lthra
2. View the most recent /var/run/log/vmkernel.log entries with the following command: tail /var/run/log/vmkernel.log
3. If the logs are not up to date, engage the hardware vendor to assist with sending an NMI to the ESXi host. This must be done before rebooting the host. Read more at Using hardware NMI facilities to troubleshoot unresponsive hosts.
4. Once the core dump has been collected, open a case with Broadcom and provide the core dump.
5. Before forcing an NMI or opening a case for deeper analysis, validate the current host configuration against the supported matrix. Incompatible drivers or firmware can often be the root cause of kernel deadlocks, especially related to the network stack.
6. Verify that the ESXi version & NIC firmware, and driver versions are compatible by checking the relevant VMware by Broadcom Compatibility Guide.
7. Power cycle the ESXi host.