No key input is accepted on the DCUI and ESXi gets into a hang up condition

Products

• VMware vSphere ESX 8.x
• VMware vSphere ESX 7.x

Issue/Introduction

An ESXi host becomes unresponsive with the following symptoms:

• An ESXi host is designated as "Not Responding" in vCenter (i.e. the vSphere Client)
• Unable to connect to the ESXi host via SSH. However, the host remains responsive to pings.
• Key input on the Direct Console User Interface (DCUI) is not accepted.
• While the ALT+F1 key functions, login attempts on the console are unsuccessful.
• Logging for services on the ESXi host stops.

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

These symptoms may be caused by the service 'vmsyslogd' hanging.

Resolution

This is a known issue, which is fixed in ESXi 7.0 U3v and ESXi 8.0 U3e.

Additional Information

If the known bug is already addressed, but the DCUI hang symptom is still observed, collect a VMKernel memory dump and DCUI memory dump by following the steps below:

1. When the DCUI and the system are running normally, connect to the ESXi shell.

2. Identify the Cartel ID (CID) of the DCUI service using the process list:
   
   ps -C | { head -1; grep dcui; }   (Note: The CID is located in the second column of the output.)

3. Verify the current configuration of the memory dump parameter. A return value of 0 indicates it is currently disabled. Substitute <CID> with the value retrieved in Step 2:
   
   vsish -e get /userworld/cartel/<CID>/debug/dumpOnKernelPanic

4. Enable the parameter so that the DCUI memory dump is included when a VMKernel memory dump is generated:
   
   vsish -e set /userworld/cartel/<CID>/debug/dumpOnKernelPanic 1

5. When the hang-up symptom is observed, send an NMI from the hardware BMC (e.g., iLO, iDRAC, XCC) to force a system panic and generate the dump.

6. After the host reboots, collect a vm-support log bundle and open a case with Broadcom Support for further investigation.