100% CPU utilization on RHEL Virtual Machine

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
Red Hat VM shows 100% CPU usage in esxtop and vCenter performance charts, while only idle CPU usage within the GuestOS tools.

Environment

Red Hat Enterprise Linux 8.4
kernel 4.18.0-305.el8 & above
VMware Virtual Machine

Cause

Constantly high due to Parameters "idle=poll" and "intel_idle.max_cstate=0"

 

• The idle=poll and max parameter does not allow the CPU to Reset, hence the CPU Utilization is constantly high
  Low Latency Performance Tuning for Red Hat Enterprise Linux 7 (Look for--> idle=poll)
• idle=poll is a way to reduce number of scheduler calls and IPIs. There is a power/cooling cost trade-off with idle=poll. This keeps processors at their maximum frequency and c-state and requires a reboot. The side-effect of this is that the CPUs may not have the thermal headroom to enter turbo frequencies which can affect performance.
• "On a physical system, idle=poll prevents the CPUs from entering power-saving states (C states) since when the CPU transitions into an idle state, a busy computation is entered. On a VM, it does nothing to control host CPU C states, but it does have the unwanted side effect of causing the host CPUs in use by the guest VM to operate at 100% capacity."

Resolution

Remove the following parameters at Guest OS Level

1. idle=poll  
   intel_idle.max_cstate=0

2. After removing these parameters, reboot the VM for the changes to take effect.
   • Disabling C-states by using a kernel command line option

Once the VM restarts, CPU utilization should return to normal.