ESXi host crashed with 'Spin count exceeded - possible deadlock with PCPU'

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi host crashed with Purple screen indicating Spin count exceeded - possible deadlock with PCPU.

Stack reports below traces:

Spin count exceeded - possible deadlock with PCPU 34
Panic: 767: Saved backtrace: pcpu 34 SpinLock spin out NMI
PShare_RemoveHint@vmkernel#nover+0xf3 stack: 0x45398f31bb88, 0xc418c7, 0x100004534c7, 0x45398f21f000, 0x4
VmMemCow_PShareRemoveHint@vmkernel#nover+0x68 stack: 0x3713024100202c54, 0x100004534c7, 0xf48, 0x45398f21f000, 0xb2326c7
VmMemRemap_Page@vmkernel#nover+0x263 stack: 0x45398f31bcbf, 0xc41800, 0xffffff00007ee400, 0xffffff000addf200, 0x10000453600
AsyncRemapProcessVMRemapList@vmkernel#nover+0x2a8 stack: 0x43167b602010, 0xc41800, 0xb232600, 0x3, 0x300406fdb
AsyncRemapProcessRemapListWrapper@vmkernel#nover+0x2d2 stack: 0x3, 0x43167b602010, 0x133defdd24bcb0e, 0x43167b602010, 0x45398f31bf9f
VmAssistantProcessTasks@vmkernel#nover+0x14b stack: 0x0, 0x0, 0x0, 0x45398f31f000, 0x453990e9f140
CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x4200358c4a60, 0x0, 0x0, 0x0
Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0

Right before the psod you might see messages regarding read error on the memory controller in /var/log/vmkernel.log file:

cpu3:2099066)MCA: 209: CE Intr G0 B7 S9c0000400####### Ad02###### M200800c0######## Pd02######/40 Memory Controller Read Error on Channel #.
cpu27:6130421)MCA: 209: CE Intr G0 B7 S9c0000400####### Ad02###### M200800c0####### Pd02######/40 Memory Controller Read Error on Channel #.
cpu27:2108941)MCA: 209: CE Intr G0 B7 S9c0000400####### Ad02###### M200800c0######## Pd02######/40 Memory Controller Read Error on Channel #.
cpu7:2107674)MCA: 209: CE Intr G0 B7 S9c0000400####### Ad02###### M200800c0######## Pd02######/40 Memory Controller Read Error on Channel #.

Stack reports NMI and lockups on other pcpus:

Spin count exceeded - possible deadlock with PCPU 40
cpu74:2107668)Panic: 767: Saved backtrace: pcpu 40 SpinLock spin out NMI
MCESafeMemCpyInt@vmkernel#nover+0x16 stack: 0x2, 0x0, 0x42003586c5d4, 0x45398829bb88, 0xffff810d02536000
MCE_SafeCall@vmkernel#nover+0x74 stack: 0x0, 0x42003586c5d4, 0x45398829bb88, 0xffff810d02536000, 0x45398829bb88
MCE_SafeMemCpy@vmkernel#nover+0x24 stack: 0xffff810d02536000, 0x1000, 0xd02535000, 0x4200359bdd7a, 0xa530400000
VmMemRemap_Page@vmkernel#nover+0x1e2 stack: 0x45398829bcbf, 0xd02400, 0xffffff0000ac0a00, 0xffffff000aff0e00, 0x10000241c00
AsyncRemapProcessVMRemapList@vmkernel#nover+0x2a8 stack: 0x431533403a40, 0xd02400, 0xb232800, 0x3, 0x300802cbe
AsyncRemapProcessRemapListWrapper@vmkernel#nover+0x2d2 stack: 0x3, 0x431533403a40, 0x133defdd23070f0, 0x431533403a40, 0x45398829bf9f
VmAssistantProcessTasks@vmkernel#nover+0x14b stack: 0x0, 0x0, 0x0, 0x45398829f000, 0x45395bb9f140
CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x4200358c4a60, 0x0, 0x0, 0x0
Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0
cpu74:2107668)Panic: 727: Halting PCPU 74.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

PSOD error stack indicates a possible hardware issue.

Resolution

• Engage the hardware vendor to run hardware diagnostic utilities.
• Validate the version of BIOS and Firmware and make sure they are compatible with the current version of esxi.