Unable to Upgrade Cluster EVC Mode from Intel skylake to Cascade Lake in vCenter Server.

Products

• VMware vCenter Server

Issue/Introduction

• When attempting to increase the Enhanced vMotion Compatibility (EVC) baseline of a cluster (e.g., from Intel "Skylake" to Intel "Cascade Lake") in VMware vCenter Server, the operation fails. The vCenter Server UI report the following error:

The host cannot be admitted to the cluster's current Enhanced vMotion Compatibility mode. Powered-on or suspended virtual machines on the host may be using CPU features hidden by that mode.

 

Environment

• VMware vCenter Server 8.0.x

Cause

This issue occurs due to persistent per-VM EVC masks embedded directly within individual virtual machine configuration (.vmx) files.

Specific entries such as featureCompat.vm.completeMasks and featMask.vm.* in vmx file prevents vCenter Server from accurately assessing the VM's EVC baseline compatibility, blocking cluster-wide EVC upgrades.

While vCenter Server tracks this information internally, it is not always exposed in the main vCenter UI "EVC CPU Mode" column, creating a discrepancy during the EVC change attempt.

Resolution

To resolve this issue, remove the per-VM EVC masks from the affected virtual machines and re-attempt the cluster EVC upgrade.

Step 1: Identify Affected Virtual Machines

• Run the EVC validation script (checkConfigureEvc.py) 
• Reveals CPU feature requirements that are not satisfied by the target EVC mode. Example output includes:

cluster: CLUSTER-XXX host: XXX vm: XXX requirements not satisfied by evcmode intel-cascadelake capabilities['req: cpuid.xcr0_master_bndcsr Num:Match:1 not met by zero (no cap)', 'req: cpuid.xcr0_master_bndregs Num:Match:1 not met by zero (no cap)']

Take snapshots of the vCenter VM. If the vCenter is in Enhanced Linked Mode (ELM), ALL VC's in ELM must be powered off together and snapshots taken when ALL are powered off, NOTE: When reverting snapshot revert all snapshots for each vCenter in ELM. For more information refer vCenter in Enhanced Linked Mode pre-changes snapshot best practice

Step 2:  Remove Per-VM EVC Masks manually or using script.

Option A - Remove Per-VM EVC Masks manually.

For each identified virtual machine, perform the following steps:

1. Power off the virtual machine.

2. Remove the per-VM EVC configuration using one of the following methods:

   • vCenter Server UI - Navigate to the virtual machine settings in the vCenter Server UI and clear/disable the per-VM EVC configuration.

   • Manual .vmx File Editing

a) Open its .vmx file for editing

b) Remove the line starting with: featureCompat.vm.completeMasks = "1"

c) Remove all lines matching the wildcard pattern featMask.vm.* 

d) Save and close the .vmx file.

• Power on the virtual machine.       

Option B - Automated Script  to remove the conflicting EVC masks -

1. Execute the script clearPerVmEvc.py on the vCenter to automatically clean up the configuration masks.
   Note - These changes made by the script will reflect on the next power cycle of the VMs. 
   
   

Step 3: Upgrade Cluster EVC Mode.

Once all per-VM EVC masks have been removed, re-attempt the cluster EVC baseline upgrade to Intel Cascade Lake in vCenter Server.