Troubleshooting ESXi PSOD Spin Count Exceeded Deadlock on PCPU 14

Products

• VMware vSphere ESXi

Issue/Introduction

This article details the troubleshooting steps for an ESXi host panic (Purple Screen of Death) presenting the error: "spin count exceeded - possible deadlock with PCPU 14".

This issue is characterized by the ESXi kernel becoming unresponsive while holding a spin lock.

Environment

VMware vSphere ESXi

 

Cause

Diagnostic analysis of the vmkernel logs indicates the deadlock is triggered by an excessive rate of System Management Interrupts (SMIs) exceeding 1,600 interrupts per second on the affected physical CPU (PCPU). This high frequency of SMIs prevents the ESXi kernel from executing necessary operations, leading to a stall in thread processing and subsequent kernel panic. 

Resolution

Please engage the HW vendor to assist in debugging this further.

1. Validate Hardware Compatibility: Verify that the server hardware and firmware versions are listed on the VMware Compatibility Guide. Ensure the system is running the manufacturer-recommended BIOS/UEFI version.
2. Update System Firmware: Check the hardware vendor portal for critical BIOS/UEFI updates specifically addressing SMI handling or CPU power management optimizations.
3. Review BIOS Settings: Consult with the hardware vendor to identify BIOS settings related to power management, C-States, or SMI handling that may be causing excessive interrupt generation.
4. Engage Hardware Vendor: If firmware updates do not mitigate the high rate of SMIs, open a support case with the hardware vendor. Provide the following information:
   • Support bundle (vm-support).
   • Timeline of the panic event(s).