ESXi host not responding due to management starvation and 100% CPU utilization

Products

• VMware vSphere ESXi

Issue/Introduction

This article addresses scenarios where an ESXi host enters a "Not Responding" state in vCenter Server despite virtual machines remaining operational. This condition occurs when the host's physical CPU resources are fully saturated, leading to a breakdown in management communication and a failure of Distributed Resource Scheduler (DRS) functions.

• ESXi host status shows as "Not Responding" in the vSphere Client.
• Host CPU utilization is observed or reported at 100% pinning.
• Virtual machines remain running but perform slowly or appear sluggish.
• DRS fails to automatically migrate (vMotion) virtual machines off the affected host. 
• In var/run/log/vmkernel.log: similar entries are observed below:

  YYYY-MM-DDTHH:MM:SS.123Z cpu24:2097152)Alert: PCPU 24: no heartbeat for 7 seconds; possible starvation or lockup.
  YYYY-MM-DDTHH:MM:SS.456Z cpu24:2097152)Warning: World 2097152 (vpxa) blocked for 12000 ms on CPU 24.
  // [Logging gap occurs here as CPU stays at 100%]
  YYYY-MM-DDTHH:MM:SS.789Z cpu0:2097200)vcpu-0:36989572: Jump in wallclock detected. Previous: 1720383022, Current: 1720384805.

Environment

• VMware vSphere ESXi

Cause

• The issue is caused by Management Starvation. When an ESXi host's CPU is pinned at 100%, the internal management agents (specifically vpxa and hostd) are starved of CPU cycles.

  1. Heartbeat Failure: vCenter expects a regular heartbeat from the vpxa agent. If the CPU is saturated, vpxa cannot send this heartbeat, causing vCenter to mark the host as "Not Responding."
  2. DRS Impairment: DRS cannot initiate vMotion because the host is unable to process the management requests required to coordinate the migration of virtual machines.

Resolution

• To restore management connectivity and allow DRS to redistribute the workload, restart the management agents on the affected host.

  1. Connect to the ESXi host via SSH or the Physical Console (ESXi Shell). 

     services.sh restart

  2.  Monitor the host in vCenter. Once the services initialize, the host should return to a "Connected" state, and DRS will begin migrating virtual machines to alleviate the CPU load.