Virtual Machine performance charts for CPU Usage metrics in VM's displays a different value in than in Guest OS

Products

• VMware vSphere ESXi

Issue/Introduction

Virtual Machine performance charts for CPU Usage metrics in VM's displays a different value in than in Guest Operating System. For example a VM will show CPU Usage MHz @ 70%  however inside vm CPU Usage in MHZ will be @ 40%.

Environment

• VMware vSphere ESXi 6.7
• VMware vSphere ESXi 6.5

Cause

1. This difference will appear when the VM have additional entries added in the virtual machines configuration file. (vmx)
   

   ethernetX.ctxPerDev = "1"
   sched.cpu.latencySensitivity.sysContexts = "<no of threads>"

2. ESXi will spawn  multiple worldID in behalf of the VM .Most of the worldID's are not CPU intensive and they will not reflect CPU usage.

3. However the worldID like sysContext and CtxPerDev are processes that help the ESXi to accelerate the packet handling.
   
   For Example:
   

   2x VCPU with 2x Nics
   
   VCPU 1 usage is  40%  and  VCPU 2 usage 40% = 40 + 40 = 80%
   Nic1 usage is 40% and Nic 2 is 20% = 40 + 20 = 60%
   Total  CPU usage = 140 / 2(vcpus) 

   
   
   

Resolution

This issue is resolved in vSphere 7.0 release.  A new counter "cpu.usage.vcpus" is added 7.0 release that sums vCPU and does not include VMX CPU usage, effectively excluding the CPU overhead.

For earlier versions,  use esxtop to determine the correct values. For more information about esxtop, please refer - Using ESXTop and Interpreting ESXTop Statistics .

Additional Information

Impact/Risks: This can trigger an alarm as the value can rise over 100% .