PSOD on host with recursive panic

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi host failed with a PSOD, the crash screen contains a backtrace similar to this example:
  

  PCPU xx: no heartbeat (0/3 IPIs received)

  PanicvPanicInt
  Panic_WithBacktrace
  Heartbeat_DetectCPULockups
  TimerWheelHandler
  Timer_BHHandler
  BH_DrainAndDisableInterrupts
  CpuSchedDispatch
  CpuSchedWait
  CpuSchedVcpuHaltWork
  VMMVMKCall_Call
  VMKVMM_ArchEnterVMKernel

  Possible hardware problem: 1 PCPU(s) [xx] did not respond to NMI
  Recursive panic (cpu xx, world xxxxxxx, depth x): ip=xxxxxxxxxxxxxx randomOff=xxxxxxxxx:
  #PF Exception 14 in world xxxxxxx:vnn0:xxxxxxx IP xxxxxxxxxxxxxx addr xxxxxxxxxxxxxx

• vmkernel core dump file (vmkernel-zdump) was not generated.

Environment

VMware vSphere ESXi 8.0.3

Cause

The system encountered recursive panic. The PSOD was caused due to PCPU didn't update pcpu heartbeat for a while and it was unresponsive to NMI. 

Resolution

• The recursive panic is usually caused by hardware problem. Please contact the hardware vendor to diagnose the hardware, especially around the PCPU.
• Updating the BIOS is also recommended.