High CPU utilization is observed in all hosts in the cluster and VM performance is affected due to this issue.
ESXi 7.x and 8.x
High CPU utilization in ESXi hosts can result from a variety of factors, both at the host and virtual machine (VM) level.
Common Causes of High CPU Utilization in ESXi Hosts include:
1. VM Workload and Overprovisioning
Running too many VMs or allocating more vCPUs than the physical CPUs (pCPUs) available can lead to high CPU contention. For example, if multiple VMs simultaneously demand high CPU, especially with a high vCPU-to-pCPU ratio (e.g., 2.5:1 or higher), the host may reach 100% CPU utilization.
Network-heavy or compute-intensive workloads on VMs can quickly saturate CPU resources, especially if resource management settings like shares or affinity are not optimized.
2. Storage and Network Related Issues
Storage path problems, such as persistent device loss (PDL) errors or misconfigured datastores, can cause ESXi hosts to repeatedly attempt access, leading to resource exhaustion and high CPU usage. Improper removal of datastores, especially those used for vSphere HA heartbeats, can trigger such issues.
Slow storage or storage queues can also force VMs to consume more CPU while waiting for I/O operations to complete, particularly if there is insufficient vRAM or if ballooning and swapping occur.
3. Host Configuration and Updates
Upgrades to ESXi can change how CPU usage is accounted for, sometimes revealing previously hidden utilization. HyperThreading (HT/SMT) configuration and changes in CPU usage accounting can make CPU usage appear higher after updates.
Power management settings, Turbo Boost, and frequency scaling can also affect how CPU usage is reported versus actual utilization.
4. Resource Scheduling and Contention
High CPU ready times indicate VMs are waiting to be scheduled on a logical processor, a sign of host CPU overload or insufficient CPU shares assigned to VMs.
Continuous vMotion activity can occur if DRS is enabled and the cluster is trying to balance workloads across hosts due to high CPU usage, further increasing overhead.
5. Background Processes and Management Services
Occasionally, management services or background processes can consume significant CPU, especially if there are issues with the management stack or if monitoring tools are left running for extended periods.
6. Application or Guest OS Issues
A runaway process or script inside a VM can cause sustained high CPU usage, which may be reflected at the host level.
High CPU utilization in ESXi hosts is often multifactorial. Effective troubleshooting requires monitoring both host and VM metrics, reviewing recent changes, and examining storage and network health.
Review Indicators/Metrics and perform Diagnostics accordingly.
1. Cause | Indicator/Metric
VM Overprovisioning | High vCPU:pCPU ratio, high ready
Storage and Network Issues | PDL errors, storage latency, driver and firmware mismatch, I/O errors
Host Updates/Config Changes | Usage spikes after upgrade
Resource Scheduling/Contention | High CPU ready, frequent vMotion
Background Services | High CPU from management daemons
VM/Application Issues | Runaway processes in guests
2. How to Diagnose High CPU Utilization
Sort VMs by CPU Usage: Use vSphere or 3rd party monitoring tools to identify which VMs are consuming the most CPU.
Check CPU Ready Times: High ready times indicate contention and scheduling delays.
Review Storage and Network Performance: Investigate storage latency, path errors, and network traffic for bottlenecks.
Monitor Host and Cluster Metrics: Look for patterns in CPU usage, utilization, and alert thresholds across hosts.
Audit Recent Changes: Consider recent upgrades, configuration changes, or storage modifications that may coincide with CPU spikes.
An effective Broadcom tool for host performance analysis is esxtop. See: Using the esxtop Utility
Log analysis can help find driver, hardware, and other errors and indicators. To collect and upload logs to Broadcom for analysis:
Collecting diagnostic information for ESXi hosts and vCenter Server using the vSphere Client