High CPU Latency %LAT_C with no CPU contention affecting VM performance

Products

• VMware vSphere ESXi

Issue/Introduction

• VM performance is severely affected
• When reviewing esxtop, %LAT_C is consistently high, with no or minimal %RDY and %CSTP
• %LAT_C is cpu latency. It denotes the percentage of time the resource pool or world was ready to run but was not scheduled to run because of CPU resource contention
  %LAT_C can be viewed in esxtop by viewing Summary stats (type 'f', 'i' and [return] in esxtop)
• %RDY is ready wait. It denotes the percentage of time the virtual machine or world was ready to run, but was not provided CPU resources
  This value also indicates that there the host has more vCPUs to PCPUs. Ideal value should be less than 10%
• %CSTP is co-stop. It denotes cpu contention due to waiting to be co-scheduled. A high %CSTP might indicate that a VM has more vCPUs than required. Ideal value should be less than 5%

Environment

VMware vSphere ESXi

Cause

High CPU latency can be caused by cpu contention, cpu ready, frequency scaling (power management/turbo)

 

Resolution

Although the host and the BIOS are configured correctly with the correct power management settings, for eg. high performance, validate frequency scaling using esxtop.

1. Launch esxtop
2. Type 'p' to switch to the power panel
3. Type 'f', 'f', and [return] to view %Aperf and %Mperf values
4. %A/MPERF column will show 100 for a core that is running on nominal frequency
5. A value higher than 100 indicates the CPU frequency has been scaled up (Turbo is enabled)
6. A value lower than 100 indicates CPU frequency has been scaled down (Power management)
7. For a performant host, %A/MPERF value should be 100 or more

To resolve the issue

• Validate power management in the BIOS and Power management settings on the ESXi host to ensure power management is not scaling down the CPU frequency
• CPU turbo should scale up and increase CPU frequency
• You may also need to check with your hardware vendor for any issues on the power supply unit 

Additional Information

Testing CPU Max Turbo Boost Frequency in ESXi

vSphere 8.0 esxtop CPU Power Panel

For a host that has performant issue, %A/MPERF value will be lower than 100 
Example:

For a performant host, %A/MPERF value should be 100 or more
Example: