ESXi Host PSOD with #PF Exception 14 in PLOGHElperQu: VMware vSAN

Products

• VMware vSAN

Issue/Introduction

VMware ESXi 8.0.x hosts experience a Purple Screen of Death (PSOD) kernel panic. This typically occurs during storage-intensive operations such as vSAN deduplication.

The PSOD screen or vmkernel-zdump backtrace includes the following markers:

• Panic Message: @BlueScreen: #PF Exception 14 in world ####:PLOGHElperQu
• Backtrace will be similar to the following:
  
  

  BlueScreen: #PF Exception 14 in world 2098576:PLOGHelperQu IP 0x############ addr 0xe0
  PTEs:0x#######################;0x##########;0x0;
  Code start: 0x42000f400000 VMK uptime: 1:12:10:43.545
  0x453a6379bd20:[0x42001120d0ae][email protected]#0.0.0.1+0x1f2 stack: 0x431a418020a8
  0x453a6379be50:[0x4200111776d0][email protected]#0.0.0.1+0x3a1 stack: 0xe3000000e3
  0x453a6379bee0:[0x420010f57816][email protected]#0.0.0.1+0x3f stack: 0x450204232698
  0x453a6379bf20:[0x420011178611][email protected]#0.0.0.1+0xb2 stack: 0x43033ea01220
  0x453a6379bf40:[0x4200111977bd][email protected]#0.0.0.1+0x5e stack: 0x453a6379f000
  0x453a6379bf60:[0x42000f55c13f]HelperQueueFunc@vmkernel#nover+0x300 stack: 0x450200002528
  0x453a6379bfe0:[0x42000fadde22]CpuSched_StartWorld@vmkernel#nover+0xbf stack: 0x0
  0x453a6379c000:[0x42000f54556f]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0
  

vmkernel.log shows hardware errors from NVMe storage devices immediately preceding the crash:

• WARNING: NvmeUtil: ###: Error on Cmd(0x####) 0x2 ... from world 0 to component "t10.NVMe____####"
• Partition: ###: Failed read for "t10.NVMe____####": I/O error
• PLOG: PLOGRegisterEventStuckIO:####: Successfully registered to detect stuck I/O

 Example PSOD: 

Environment

• VMware ESXi 8.0.x
• VMware vSAN with Deduplication and Compression enabled
• NVMe Storage Devices

Cause

A race condition exists in the PLOG component when an underlying NVMe drive encounters hardware-level read/write failures concurrently with a vSAN deduplication process and other storage operations at the same time.

Resolution

Broadcom engineering is aware of this behavior and reviewing a fix for a future release.
The issue is triggered by a race condition during a hardware failure.

1. Review the vmkernel logs to identify the specific NVMe device reporting I/O errors or NvmeUtil warnings.
2. Engage the hardware vendor to perform high-stress diagnostics on the identified NVMe drive.
3. Replace the faulty NVMe hardware.
4. Once replaced, monitor the host during the vSAN resync