Determining if multiple virtual CPUs are causing performance issues

Products

• VMware
• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

This article helps you determine if multiple virtual CPUs (vCPUs) are causing performance issues.

Symptoms:

You may experience these performance issues with a multiple CPU virtual machine running on an ESX host:

• Poor transfer speeds when copying data to or from a virtual machine
• Backup jobs time out or are very slow
• A virtual machine performs poorly

Environment

^{VMware vSphere Esxi 5.x}

^{VMware vSphere Esxi 6.x}

^{VMware vSphere Esxi 7.x}

Cause

<sup>A virtual machine can have multiple virtual CPUs. The scheduler of the ESXi needs to place all of these in the the same processing cycle of the physical CPU. This only works if that cycle has enough threads available to place the virtual CPUs. If there aren't for example because other virtual machines are already utilizing a part of the physical threads, the VM has to wait for the required resources to be available again. 

Even when only a part of the virtual CPUs is really required for the current workload in the virtual machine, that workload has to wait until all, even the currently unused virtual cores, can be run on the physical processor. This behavior is also known as "co-stop"

This placement becomes more difficult, the higher the number of virtual CPUs for a specific virtual machine is, and it can lead to measurable performance loss for the guest operation system or applications within this VM.</sup>

Resolution

To determine if multiple vCPUs assigned to your virtual machine is causing poor performance:

1. Open a console prompt on the ESX host or initiate an SSH connection to it. For more information, see Opening a command or shell prompt (1003892).
2. Type esxtop and press Enter.
3. In the CPU screen, check the %CSTP value. If this number is higher than 3.00, the performance issues may be caused by the vCPU count. Try lowering the vCPU count of the virtual machine by 1.
   
   Note: The %CSTP value represents the amount of time a virtual machine with multiple virtual CPUs is waiting to be scheduled on multiple cores on the physical host. The higher the value, the longer it waits and the worse its performance. Lowering the number of vCPUs reduces the scheduling wait time.

To lower the vCPU count:

Note: The virtual machine must be powered off to perform these steps.

1. Right-click on the virtual machine and click Edit Settings.
2. Click CPUs.
3. Use the Number of virtual processor drop-down to lower the vCPU count by 1.
4. Click OK.
5. If your virtual machine still experiences performance issues, and if its kernel or HAL can handle switching to a single vCPU, lower the vCPU count to 1.
   
   Warning: If your virtual machine's kernel or HAL cannot handle switching to a single vCPU, unexpected behaviour may occur.