ESXi PSOD with error "BlueScreen: CPU X/ World XXXXX tried to re-acquire lock" "PsaNVMe_AsyncTokenIODone"

Products

• VMware vSphere ESXi 8.0
• VMware vSphere ESX 8.x
• VMware vSphere ESX 7.x

Issue/Introduction

You may see a similar PSOD screen.

Or similar traces in the vmkernel.log

yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)Backtrace for current CPU #0, worldID=XXXXX, fp=0x0
yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)0x453acd51bcc0:[0x42000ab7bbc0]PanicvPanicInt@vmkernel#nover+0x20c stack: 0x0, 0x42000ab7bbc0, 0x0, 0x420000000001, 0x42000ab7bbc0
yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)0x453acd51bd70:[0x42000ab7c33c]Panic_NoSave@vmkernel#nover+0x4d stack: 0x453acd51bdd0, 0x453acd51bd90, 0x453acd51bdd0, 0x0, 0xfa53e4
yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)0x453acd51bdd0:[0x42000aa28982]LockCheckSelfDeadlockInt@vmkernel#nover+0x9f stack: 0x200000000, 0x6c3db0f2aa0da0, 0x1, 0x45e13545c300, 0x45e13545c300
yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)0x453acd51bde0:[0x42000ab852bd]SP_WaitLock@vmkernel#nover+0x16a stack: 0x1, 0x45e13545c300, 0x45e13545c300, 0x1, 0x430b1dd8e480
yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)0x453acd51be20:[0x42000ab852f0]SPLockWork@vmkernel#nover+0x29 stack: 0x45e12e6a32c0, 0x430b1dd8e400, 0x45e12e6a32c0, 0x45e13545c300, 0x430b1dd8fe90
yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)0x453acd51be30:[0x42000b0c1df9]AsyncPopCallbackFrameInt@vmkernel#nover+0x1e stack: 0x45e12e6a32c0, 0x45e13545c300, 0x430b1dd8fe90, 0x42000aebc69a, 0x45da8197ef40
yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)0x453acd51be60:[0x42000aebc699]PsaNVMe_AsyncTokenIODone@vmkernel#nover+0x76 stack: 0x430b1dd8e400, 0x430b1dd8e400, 0x45e12e6a32c0, 0x45e12e6a3480, 0x430b1dd8fe90
yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)0x453acd51bea0:[0x42000aeb84ad]PsaNvmeDeviceTimeoutHandlerFn@vmkernel#nover+0x3b2 stack: 0x100000004, 0x6c3d5e046d87f0, 0x41ffcaec9f40, 0x7, 0x4200418016c0
yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)0x453acd51bf60:[0x42000af1d029]PsaStorDeviceTimeoutHandlerFn@vmkernel#nover+0x62 stack: 0x0, 0x420000000cd7, 0x430b1dd8e400, 0x10, 0x14
yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)0x453acd51bfa0:[0x42000afc2a9f]PsaStorTaskMgmtWorldFunc@vmkernel#nover+0x8c stack: 0x453a5469f100, 0x453acd51f100, 0x0, 0x0, 0x0
yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)0x453acd51bfe0:[0x42000b0d67b2]CpuSched_StartWorld@vmkernel#nover+0xbf stack: 0x0, 0x42000ab44cf0, 0x0, 0x0, 0x0
yyyy-mm-ddThh:mm:52.565Z cpu0:16405476)0x453acd51c000:[0x42000ab44cef]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0
yyyy-mm-ddThh:mm:52.581Z cpu0:16405476)VMware ESXi 8.0.3 [Releasebuild-24585383 x86_64]

Environment

VMware ESXi 7.0

VMware ESXi 8.0

Cause

Based on the memory dump, the PSOD occurred when a thread attempted to acquire a lock that was already held by PCPU 0. The issue was triggered following an I/O stall on the NVMe device.

Resolution

Fixed in ESXi version:  7.0.3 P10 and  8.0.3.0 P05.