ESXi host fails with a PSOD "no heartbeat" caused by an incompatible driver

Products

• VMware vSphere ESXi

Issue/Introduction

PSOD Backtrace:

YYYY-MM-DDTHH:MM:SSZ cpu##:######)@BlueScreen: PCPU ##: no heartbeat (2/3 IPIs received)
YYYY-MM-DDTHH:MM:SSZ cpu##:######)Code start: 0############### VMK uptime: 0:00:08:59.077
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]PanicvPanicInt@vmkernel#nover+0#### stack: 0#############
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]Panic_WithBacktrace@vmkernel#nover+0### stack: 0#############
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]Heartbeat_DetectCPULockups@vmkernel#nover+0#### stack: 0#26e8
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]TimerWheelHandler@vmkernel#nover+0### stack: 0#############
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]Timer_BHHandler@vmkernel#nover+0### stack: 0#############
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]BH_DrainAndDisableInterrupts@vmkernel#nover+0### stack: 0#0
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]IntrCookie_VmkernelInterrupt@vmkernel#nover+0### stack: 0####
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]IDT_IntrHandler@vmkernel#nover+0### stack: 0#0
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]gate_entry@vmkernel#nover+0### stack: 0#0
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]Power_ArchPerformWait@vmkernel#nover+0#### stack: 0############
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]Power_ArchSetCState@vmkernel#nover+0### stack: 0##
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]CpuSchedIdleLoopInt@vmkernel#nover+0#### stack: 0##############
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]CpuSched_IdleLoop@vmkernel#nover+0### stack: 0####
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]Init_SlaveIdle@vmkernel#nover+0### stack: 0###
YYYY-MM-DDTHH:MM:SSZ cpu##:######)0##########:[0#############]SMPSlaveIdle@vmkernel#nover+0#### stack: 0###
YYYY-MM-DDTHH:MM:SSZ cpu##:######)base fs=0## gs=0############# Kgs=0##

Note: The backtrace is only an example.

Environment

VMware vSphere ESXi 7.0 U3

VMware vSphere ESXi 8.0 U3

Cause

• PSOD happened because PCPU ### got hung and was unresponsive even to NMI. This looks like a platform hardware issue.

Resolution

Running incompatible driver or firmware versions can lead to system instability and unpredictable behavior. 

Please perform the following steps to bring the hosts into a supported state:

1. Update Server Firmware / BIOS

• Dell PowerEdge R7625 (ESXi 7.0 U3): Installed Firmware 1.4.6 is not listed on the compatibility guide. View Server Compatibility Guide

• Dell PowerEdge R840 (ESXi 8.0 U3): Installed BIOS 2.24.0 is currently outdated by two releases. It is highly recommended to update the BIOS to a version aligned with your ESXi release. View Server Compatibility Guide

2. Update Network Drivers

• vmnic2 & vmnic3 (ESXi 7.0 U3): Installed Driver version 216.0.50.0 is not supported for this ESXi version. Please update to a supported driver. View I/O Compatibility Guide / View I/O Compatibility Guide for R840

3. Hardware Health Check Once the updates are complete, we strongly suggest engaging your hardware vendor (Dell) to verify the overall health of the hardware platform and ensure no underlying hardware faults remain.

Additional Information

This is only an example, for more information about Compatibility, please visit:

https://compatibilityguide.broadcom.com