Using Virtual CPU Performance Monitoring Counters

Products

• VMware Desktop Hypervisor
• VMware vSphere ESXi

Issue/Introduction

CPU Performance Monitoring Counters (PMCs) provide a way for software to monitor and measure processor performance. These counters are commonly used by tools such as software profilers. Beginning with virtual machines that have ESX 5.1 and later compatibility (hardware version 9), you can enable the virtual performance monitoring counters (vPMCs) feature to allow software running inside virtual machines to access this performance information, just as it would when running on a physical machine.

Minimum Requirements

• Intel Nehalem Generation or later processors, or AMD Opteron Generation 3 (Greyhound) or later processors.
• Intel VT-x or AMD-V must be enabled in the BIOS so that hardware assisted virtualization is possible.

Virtual Performance Counter Model Specific Registers (MSRs)

When Virtual CPU Performance Monitoring Counters is enabled, the following MSRs are virtualized and available to the guest operating system.

Intel CPUs

• IA32_PERFEVTSELx
• IA32_PMCx
• IA32_FIXED_CTRx
• IA32_PERF_GLOBAL_CTRL
• IA32_PERF_GLOBAL_STATUS
• IA32_PERF_GLOBAL_OVF_CTRL
• IA32_FIXED_CTR_CTRL

AMD CPUs

• PERF_CTLx
• PERF_CTRx
  

Environment

VMware vSphere ESXi
VMware Workstation Pro
VMware Fusion Pro

Resolution

The virtual PMCs are accessed in the same way as the PMCs of the underlying physical CPU. See the Intel 64 and IA-32 Architectures Software Developer's Manual and the AMD BIOS and Kernel Developer's Guide (BKDG) for details.

Enabling Virtual CPU Performance Monitoring Counters

vSphere 7.0 - Activate Virtual CPU Performance Counters

vSphere 8.0 - Activate Virtual CPU Performance Counters

VMware Workstation Pro: Configuring Virtual Machine Processor Settings

VMware Fusion Pro: Set Advanced Processor Options

Event Counting

The virtual PMCs can count the same processor core events as the underlying physical CPU's PMCs. However, because the virtual CPUs of a virtual machine can be descheduled for periods of time and some guest instructions might be emulated by the hypervisor rather than retired directly by the physical CPU, the PMCs are not passed through directly to the virtual machine. Instead, access to the PMC is through the hypervisor. You can control the virtual PMC event counting behavior by setting the following virtual machine configuration options in the .vmx file:

 

Option	Description
vpmc.freezeMode=hybrid	This is the default behavior. The instructions retired and branches retired events count guest instructions only. When the CPU executes hypervisor instructions, these events do not increment. All other events increment whenever the physical CPU executes either guest or hypervisor instructions on behalf of the virtual machine. When a virtual machine is not scheduled on a physical CPU, its virtual PMCs always stop incrementing. This way, metrics involving ratios of guest instructions can be used to calculate the cost of executing instructions on the virtual machine. For example, instructions per cycle (IPC) indicate the average number of guest instructions that can be retired per cycle that the physical CPU is used to execute the virtual machine.
vpmc.freezeMode=guest	All events increment only when guest instructions are directly executing on a physical CPU. When the hypervisor code is executing, no events increment.
vpmc.freezeMode=vcpu	All events increment while a physical CPU is being used to execute guest code or hypervisor code on behalf of the virtual machine. Events stop incrementing only while the virtual machine is descheduled.

 

Available Performance Monitoring Counters

In some cases, a performance monitoring counter cannot be virtualized. This occurs when the host is already using the physical CPU performance counter for another use, as in the following cases:

• The host BIOS might use one or more PMCs.
• On ESX, the VMkernel might use a PMC for internal use.
• On Workstation and Fusion, the host OS or a profiler running on the host might use one or more PMCs.
• The Fault Tolerance feature might use one or more PMCs.

In general, if a physical CPU PMC is in use, the corresponding virtual CPU PMC is not functional and is unavailable for use by the guest. Guest OS software detects unavailable general purpose PMCs by checking for a non-zero event select MSR value when a virtual machine powers on. The guest software detects unavailable fixed-function PMCs by checking for a corresponding non-zero enable bit in the IA32_FIXED_CTR_CTRL MSR.

 

If the host is using the following fixed-function performance counters, that virtual PMC is emulated:

• unhalted core cycles
• unhalted reference cycles

When the cycles counter is emulated, it is functional only when programmed to count in both OS and user modes. Also, an emulated core cycles counter might count reference cycles rather than core cycles.

 

vMotion and Checkpoint Compatibility

When the virtual PMC feature is enabled, additional CPU compatibility checks are performed before a virtual machine can be migrated between hosts. The performance monitoring events of the source and destination CPUs must be compatible. The list of performance monitoring events must be the same on the source and destination CPUs. In general, two CPUs have compatible performance monitoring events if they belong to the same microarchitecture. See the Intel 64 and IA-32 Architectures Software Developer's Manual and the AMD BIOS and Kernel Developer's Guide (BKDG) for the list of performance monitoring events for each CPU. In addition, to successfully migrate a virtual machine, all counters that are available on the source CPU must also be available on the destination CPU.

Enhanced vMotion Compatibility

You cannot enable Virtual Performance Monitoring Counters in a virtual machine that is using Enhanced vMotion Compatibility.