Allowing vCPUs of an SMP virtual machine to share an HT-enabled core might improve performance

Products

• VMware vSphere ESXi

Issue/Introduction

HyperThreading (HT) technology is adopted again with recent Intel processors. Typically, two logical processors are presented per physical core. As these two logical processors compete for the same pipeline in a core, the computational capacity of one logical processor is less than that of a core. Utilizing both logical processors provides 10% - 30% higher throughput than when only one logical processor is used. Therefore, the capacity of each logical processor when both processors are busy is about 55% - 65% of a single logical processor running alone on the core.

To optimally utilize CPU capacity on a system, the CPU scheduler prefers a fully idle core to a partially idle core when it schedules a vCPU. Here, a fully idle core means that both logical processors on the core are idle while a partially idle core means that only one of the logical processors is idle.

The co-scheduling algorithm avoids scheduling a vCPU of an SMP virtual machine to a processor on which a sibling vCPU is running (or queued). This prevents sibling vCPUs from competing for the same processor because this results in higher costop and ready time.

Co-scheduling on an HT processor extends this notion further by considering the logical processor as above but also by looking at the sibling logical processor to avoid scheduling sibling vCPUs of an SMP virtual machine on the same physical core. The scheduler thus tries to equalize the resources each vCPU gets as well as to maximize the overall performance of the virtual machine.

Problem:

The policy of not allowing sibling vCPUs of an SMP virtual machine to share a core can be restrictive and might artificially cap CPU utilization and yield sub-optimal performance. This is most noticeable when the number of vCPUs equals the number of logical processors since no processors would be eligible for scheduling choices. The problem has also been seen for a smaller number of vCPUs.

Environment

VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded
VMware ESX 4.1.x

Resolution

This issue is applicable only to the ESX 4.1 Update 1 release. Disabling the current vCPU related policy allows vCPUs from the same virtual machine to share two logical processors of one core, which leads to more scheduling choices and higher CPU utilization.

 

To disable the policy:

• For ESX, run this command on the service console:
  
  esxcfg-advcfg ––set 0 /Cpu/DontShareCoreWithinVM
  
   
• For ESXi, run this command on a Windows or Linux virtual machine with vSphere CLI installed:
  
  vicfg-advcfg ––set 0 /Cpu/DontShareCoreWithinVM

Caution: DontShareCoreWithinVM is an advanced parameter and should only be changed under guidance from VMware Support or after thorough testing in the your environment.