VMs fail to vMotion to ESXi 6.7 P03 hosts with error " A performance counter used by the guest is not available on the host CPU."

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
VPMC-enabled VMs may fail to vMotion to a host returning the error. "A performance counter used by the guest is not available on the host CPU"

A VPMC-enabled VM will have the following in its VMX

       vpmc.enable = "TRUE"

Similar entries are seen in the vmware.log file
2020-10-14T16:13:26.807Z| vmx| I125: MonPmc: ctrBase 0x4c1 selBase 0x186/1 PGC 1/1 SMM 1 drain 0 AMD 0
2020-10-14T16:13:26.807Z| vmx| I125+ MonPmc: gen counters num: 4 width 48 write width 48
2020-10-14T16:13:26.807Z| vmx| I125+ MonPmc: fix counters num: 3 width 48; version 4
2020-10-14T16:13:26.807Z| vmx| I125+ MonPmc: unavailable counters: 0x200000008
2020-10-14T16:13:26.807Z| vmx| I125: CPT: Restoring checkpoint (null)
2020-10-14T16:13:26.807Z| vmx| I125: Migrate_Open: Restoring from <x.x.x.x> with migration id 7284075753188080409
2020-10-14T16:13:26.807Z| vmx| I125: DUMPER: Restoring checkpoint version 8.
2020-10-14T16:13:26.807Z| vmx| I125: MigrateSetStateFinished: type=2 new state=12
2020-10-14T16:13:26.807Z| vmx| I125: MigrateSetState: Transitioning from state 11 to 12.
2020-10-14T16:13:26.808Z| vmx| I125: Migrate: Caching migration error message list:
2020-10-14T16:13:26.808Z| vmx| I125: [msg.checkpoint.migration.failedReceive] Failed to receive migration.
2020-10-14T16:13:26.808Z| vmx| I125: [msg.vpmc.unavailcounters] A performance counter used by the guest is not available on the host CPU.

VMs can successfully vMotion to other hosts that are not running P03

VMs without  VPMC enabled are not impacted.

Environment

VMware vSphere 6.7.x
VMware vSphere ESXi 6.7

Cause

An Intel microcode update was released changing the behavior of the PMU of certain CPUs[1].  The fix utilizes the PMC3 counter,  leaving it unavailable for normal use.  ESXi 6.7 P03 (build 16713306) detects the presence of this microcode fix and marks the PMC3 counter as 'in-use' preventing VMs from using this counter.

This causes the above issue when a VM with VPMC enabled was powered on a host running a build older than 6.7 P03.   If such a VM attempts to vMotion to a host running 6.7 P03 or higher, the counter is no longer available for use causing the vMotion to fail leaving the VM running on the original host.  The VMs are able to successfully vMotion to other hosts running < P03

A VM that was powered on a P03 or higher host is capable of vMotioning to an older host but would not be able to return via vMotion.


[1] Intel® Xeon® Processor E3 v5 and v6 Family (codename Skylake, Kaby Lake)
    Intel® Xeon® D (code name Skylake-D)
    Intel® Xeon® Scalable Processor and 6th, 7th, and 8th Generation Intel® Core™ i7 and i5 (code name Skylake, Kaby Lake, Coffee    Lake and Whiskey Lake)

Resolution

There is not solution to this problem.  This is currently performing as designed.  

Workaround:
VMs must be powered down and cold-migrated to P03+ hosts.  Once the VM has been cold-migrated and powered on a P03 host, it will be able to successfully vMotion between all other P03+ hosts.

Additional Information

Intel errata of the PMU change can be found here - http://cdrdv2.intel.com/v1/dl/getContent/604224