Workloads perform poorly on ESX SMP virtual machines

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Workloads that rely on inter-process communication perform poorly on ESX SMP virtual machines when compared to physical machines
• esxtop and VirtualCenter Performance Counters indicate that:
• • The CPU utilization does not reach 100% across all virtual CPUs
  • There is a lot of idle time
    
    Note: You see similar results when reviewing the utilization within the Guest.
    
• There are a higher than usual amount of SWTCH/s (World Switches per second) and WAKE/s (World wakeups per second) as seen in the esxtop CPU window with EVENT COUNTS/s added.
  
  Note: The value is typically below 100.

• This issue has been observed in Windows 2003 64bit guests but may also occur in other operating systems

Environment

VMware vSphere ESXi 5.0
VMware ESX Server 3.5.x
VMware vSphere ESXi 5.5
VMware ESX 4.0.x
VMware vSphere ESXi 5.1

Resolution

The guest operating system issues a Halt instruction which stops (or de-schedules) the vCPU on the ESX host. Keeping the virtual machine spinning longer before Halt negates the number of inter-processor Wake up requests.

Setting CPU Affinity on the processes that are performing badly within the Guest improves performance, as does running the jobs in a single CPU virtual machine. However, these options may not be suitable in your environment.

Note: A single CPU may not be suitable because there are times when the Guest needs more CPU.

To resolve this issue, try setting the value of virtual machine configuration option monitor.idleLoopMinSpinUS = <n> to less than 2000 microseconds.

To set the value of monitor.idleLoopMinSpinUS to less than 2000 microseconds:

1. Schedule some down time.
2. Power off the virtual machine.
3. Add the following options to the configuration file of the virtual machine (.vmx):
   
   Note: For instructions on how to edit the .vmx file, see Tips for editing a .vmx file (1714).
   
   monitor.idleLoopSpinBeforeHalt = true
   monitor.idleLoopMinSpinUS = <n>
   
   where <n> is a value less than 2000 (for example, 100 or 250 microseconds)
   
   Note: Avoid setting monitor.idleLoopMinSpinUS to high values, as this causes the virtual machine to spin longer in an idle loop. This causes differences in CPU utilization as reported within the Guest and as reported by ESX. Setting a high value means that the vCPUs are using Physical CPU resource time while idling. The fact that the vCPUs are idling more becomes significant in environments where there are high ratio of virtual machines per ESX host, especially considering that not all workloads gain from this setting.
   
4. Power on the virtual machine.