Understanding the CPU ready values in the vSphere Client advanced performance charts

Products

• VMware vSphere ESXi

Issue/Introduction

The way CPU ready values are displayed in the vSphere Client performance charts is a bit difficult to understand. This article aims to provide explantions for what the charts actually show and how it should be interpreted.

Environment

vCenter Server 6.x

vCenter Server 7.0.x

vCenter Server 8.0.x

Resolution

What is CPU ready?

CPU ready shows the amount of time a virtual machine (or rather its virtual CPU cores spent in ready state, that is a state where the vCPUs were waiting to be scheduled in the threads of the physical processor of the server running the hypervisor software (like ESXi), the host. This can be reported as a percentage value (like %RDY in esxtop) or as an absolute value like the vSphere Client performance charts do.

 

What is so special about CPU ready?

Other than a lot of the other performance counters, CPU ready is not a momentary value, but will always be an accumulation over time, since as mentioned about it shows how much time the virtual CPUs were in the state of waiting to be scheduled. In esxtop the %RDY counter shows a percentile value that is calculated based on the time since the last sample was taken.

In case of the performance charts in vSphere Client however an absolute number in milliseconds is shown. This value means nothing in itself though, it can only be evaluated against the interval between the data points in the chart.

 

About intervals

None of the charts in vSphere client shows seamless data. Not matter which one you are looking at, it will always have distances between the actual points of data, which is the statistics interval.

These intervals are different, depending on the range the chart is displaying.

• When the chart is set to "Real-Time", the data points will be 20 seconds apart.
• When looking at "Last Day", they will be once every 5 minutes, as configured in the vCenter Statistics settings
• When looking at any data older than 24 hours, but younger than 7 days, the distance between the data points will be 30 minutes (again as configured in the vCenter Statistics settings), for data between 7 days and a month this will be 2 hours, and for data between 1 month and 1 year old the points in the chart will be 1 day apart of each other.

This screenshot shows the default intervals for any data outside other than "Real-Time":

You can change these values, but always keep in mind that changing them to lower values will introduce a lot of additional strain to the performance of the vCenter database and thus on vCenter itself, and will increase the space requirements for the SEAT database massively. Lowering the interval durations is therefore not recommended.

 

Why does this matter? 

Because the intervals are different, and because CPU ready is summed up over time, you cannot directly compare the value for this counter between different charts.

For example, let's say you look at "Real-Time" and see a value of 500 for CPU ready. Now you look at the "Last Day" chart, and suddenly the value is 7500 milliseconds. D

Does this mean that the virtual CPUs spent more time in ready state yesterday than it does today?

No it does not, because right now the interval is 20 seconds, and for yesterday an interval of 5 minutes (aka 300 seconds) needs to be taken into account. To get comparable values, you need to look at the actual percentages instead.

For the value in "Real-Time" this is:

50 ms * 100 / 20;000 ms = 0.025 percent

For "Last Day" it is:

7500 ms * 100 / 300,000 ms = 0,025 percent

So basically both charts are showing an equal situation, both time the virtual CPUs spent a 40th of a percent in ready state - nothing changed in the meantime. Now, these are theoretical values, in practice they will most likely be higher, but the point should be clear.

 

What else is important to know?

One other thing that has not been mentioned above, but which will matter for practical observations is the fact that both esxtop and the performance charts do show CPU ready and %RDY as a sum for all of the virtual cores of a VM. 

Mathematically this provides good results when compared to the overall number of available threads on the physical CPUs in the hypervisor-host. For performance analyses it can be a bit misleading though, as it does not reflect the time the actual virtual machine spent ready. As ESXi (like other hypervisors) can only ever schedule all virtual CPUs of a virtual machine in the same processor-cycle of the physical CPU, you will need to divide the value shown in the chart by the number of virtual CPUs the VM has, to find out how much of its time the VM was waiting to be scheduled.

Going by the example above with the chart showing 500 ms in "Real-Time" mode, let us assume that the virtual machine has 4 virtual cores.

The maths would in this case be:

(50 ms * 100 / 20;000 ms) / 4 = 0.00625 percent 

The virtual machine in this example spent 0.00625 percent of its run time in ready state, or a 160th of a percent.

 

Additional Information

Please keep in mind that CPU ready or %RDY in itself is not an indicator for existing performance issues. A virtual machine can spend a lot of time in ready state without ever running into any slowness/bottlenecks or other performance problem. Therefore CPU ready should not be taken on its own, but instead should be used as a tool, or better one of the tools used when troubleshooting actual performance issues.