ESX host is unresponsive in vCenter and at the ESX UI, but has not crashed with a PSOD

Products

• VMware vSphere ESXi

Issue/Introduction

When this type of issue occurs, it is possible to experience one or more of the following symptoms:

• The ESX host is no longer responding in vCenter.
• The ESX UI is inaccessible.
• The ESX host is not responding to DCUI input / requests.
• At times, the VMs managed by the host may become unresponsive as well.  
• A hard / cold reboot is required to recover the ESX host.
• All logging for the host stops at / around the same time (vpxd, hostd, vmkernel, vobd, etc.).

Environment

ESX 8.x

Cause

When an ESX host becomes unresponsive and stops logging, this is normally a sign of an underlying hardware issue.  

You may see entries in the IPMI logs similar to the following around the time of the stoppage: 

Record:577:
   Record Id: 577
   When: ####-##-##T##:##:##
   Event Type: 126 (Unknown)
   SEL Type: 2 (System Event)
   Message:
   Sensor Number: 146
   Raw:
   Formatted-Raw: 41 02 02 83 f7 00 69 20 00 04 0c 92 7e 20 03 34

 

In the above example, the sensor ID correlates to memory DIMM: 

Node-Sensor  Description
0.146        Memory Module 27 DDR5_P2_F1_ECC           

   

Resolution

A case should be open with both Broadcom Support and with the hardware vendor for a full review of the issue.