VMware EVC and CPU Compatibility FAQ
search cancel

VMware EVC and CPU Compatibility FAQ

book

Article ID: 313545

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

This document provides answers to common questions on Enhanced vMotion Compatibility (EVC) and CPU compatibility.

Environment

VMware vCenter Server

VMware vSphere ESXi

Resolution

What is EVC?

EVC is short for Enhanced vMotion Compatibility. EVC allows virtual machines to migrate between different generations of CPUs made by the same CPU vendor.

What is the benefit of EVC?

EVC allows VMs to migrate between different generations of CPUs. Older and newer server generations using different CPU generations in the same cluster are able to migrate virtual machines with vMotion between these hosts when EVC is enabled. This makes adding new hardware into existing infrastructure easier and helps extend the value of existing hosts. With EVC, full cluster upgrades can be achieved with no virtual machine downtime whatsoever. As new hosts are added to the cluster, VMs can be migrated with vMotion without VM workload downtime to the new hosts and the older hosts can be retired.

How do I use EVC?

EVC is enabled for a cluster in vCenter Server inventory. After it is enabled, EVC ensures that migration with vMotion is possible between any hosts in the cluster. Only hosts that preserve this property can be added to the cluster.

How does it work?

After EVC is enabled, all hosts in the cluster are configured to present the CPU features of a user-selected processor type to all virtual machines running in the cluster. This ensures CPU compatibility for vMotion even though the underlying hardware might be different from host to host. Identical CPU features are exposed to virtual machines regardless of which host they are running on, so that the virtual machines can migrate between any hosts in cluster.

Can migrations with vMotion still fail for virtual machines within an EVC cluster?

Yes. More factors than CPU compatibility are used to determine vMotion compatibility. For example, if the virtual machine does not reside on storage shared by the source and destination hosts, then migration with vMotion cannot occur between these hosts.

Does EVC allow AMD and Intel CPUs to be vMotion compatible?

No. An EVC-enabled cluster only allows CPUs from a single vendor to be used in the cluster. vCenter Server does not allow a host with a different CPU vendor to be added into an EVC-enabled cluster.

What EVC modes are available?

vCenter 6.5 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
AMD Opteron "Steamroller" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation
Intel "Broadwell" Generation

vCenter 6.7 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
AMD Opteron "Steamroller" Generation
AMD "Zen" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation
Intel "Broadwell" Generation
Intel "Skylake" Generation

vCenter 7.0 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
AMD Opteron "Steamroller" Generation
AMD "Zen" Generation
AMD "Zen 2" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation
Intel "Broadwell" Generation
Intel "Skylake" Generation
Intel "Cascade Lake" Generation

vCenter 7.0 U2 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
AMD Opteron "Steamroller" Generation
AMD "Zen" Generation
AMD "Zen 2" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation
Intel "Broadwell" Generation
Intel "Skylake" Generation
Intel "Cascade Lake" Generation
Intel "Ice Lake" Generation

vCenter 8.0 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
AMD Opteron "Steamroller" Generation
AMD "Zen" Generation
AMD "Zen 2" Generation
AMD "Zen 3" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation
Intel "Broadwell" Generation
Intel "Skylake" Generation
Intel "Cascade Lake" Generation
Intel "Ice Lake" Generation

vCenter 8.0 U1/U2 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
AMD Opteron "Steamroller" Generation
AMD "Zen" Generation
AMD "Zen 2" Generation
AMD "Zen 3" Generation
AMD "Zen 4" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation
Intel "Broadwell" Generation
Intel "Skylake" Generation
Intel "Cascade Lake" Generation
Intel "Ice Lake" Generation
Intel "Sapphire Rapids" Generation

Which CPUs are compatible with each EVC mode?

To determine the EVC modes compatible with a CPU, search the VMware Compatibility Guide for the server model or CPU family, and select the entry in the CPU Series column to display the compatible EVC modes.

What are the differences between the modes?

Each mode corresponds very closely to the features available in processors with the same name. Newer processors and their corresponding modes include additional features such as new instructions. For example, the Intel Xeon Core i7 processor added the SSE4.2 instruction set. These instructions are not available in earlier processors or their corresponding EVC modes.

AMD EVC modes expose these features:

EVC Mode Available Features
AMD Opteron Generation 1 All features of AMD Opteron Rev. E CPUs
AMD Opteron Generation 2 All features of AMD Opteron Generation and additional CPU features including CMPXCHG16B and RDTSCP
AMD Opteron Generation 3 All features of AMD Opteron Generation 2 and additional CPU features including SSE4A, MisAlignSSE, POPCOUNT, ABM (LZCNT)
AMD Opteron Generation 3 (no 3Dnow!) Applies baseline feature set of AMD Opteron Generation 3 (Greyhound) processors, with 3DNow! support removed, to all hosts in the cluster.

This mode allows clusters containing AMD hosts to accept AMD processors without 3DNow! support.
AMD Opteron Generation 4 Applies baseline feature set of AMD Opteron Generation 4 (Bulldozer) processors to all hosts in the cluster.

This EVC mode exposes additional CPU features including SSSE3, SSE4.1, AES, PCLMULQDQ, XSAVE, AVX, XOP and FMA4.
AMD Opteron "Piledriver" Generation Applies baseline feature set of AMD Opteron "Piledriver" Generation processors to all hosts in the cluster.

This EVC mode exposes additional CPU features including FMA3 (three operand FMA, aka Intel's FMA), BMI1 and TBM.
AMD Opteron "Steamroller" Generation Applies baseline feature set of AMD Opteron "Steamroller" Generation processors to all hosts in the cluster.

This EVC mode exposes additional CPU features including XSAVEOPT and FSGSBASE.
AMD "Zen" Generation Applies baseline feature set of AMD "Zen" Generation processors to all hosts in the cluster.
This EVC mode exposes additional CPU features including XSAVEOPT, RDFSBASE, RDGSBASE, WRFSBASE, WRGSBAS and FSGSBASE.
AMD "Zen 2" Generation Applies baseline feature set of AMD "Zen 2" Generation processors to all hosts in the cluster.
This EVC mode exposes additional CPU features including CLWB, UMIP, RDPID, XGETBV with ECX = 1, WBNOINVD, and GMET.
AMD "Zen 3" Generation Applies the baseline feature set of AMD "Zen 3" Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including always serializing LFENCE, INVPCID, PSFD, SSBD, PCID, PKU, VAES, VPCLMULQDQ, and shadow stacks.
AMD "Zen 4" Generation

Applies the baseline feature set of AMD "Zen 4" Generation processors to all hosts in the cluster. Compared to the AMD "Zen 3" Generation EVC mode, this EVC mode exposes additional CPU features including Fast Short CMPSB and STOSB, Automatic IBRS, AVX512BF16, AVX512BITALG, AVX512BW, AVX512CD, AVX512DQ, AVX512F, AVX512IFMA, AVX512VBMI, AVX512VBMI2, AVX512VL, AVX512VNNI, AVX512VPOPCNTDQ, GFNI, IBRS, and Upper Address Ignore.


Intel EVC modes expose these features:

EVC Mode Available Features
Intel "Merom" Generation (Intel Xeon Core2) All features of Intel Core2 CPUs
Intel "Penryn" Generation (Intel Xeon 45nm Core2) All features of Intel Core2 CPUs and additional CPU features including SSE4.1
Intel "Nehalem" Generation (Intel Xeon Core i7) All features of Intel Core2 CPUs and additional CPU features including SSE4.2 and POPCOUNT
Intel "Westmere" Generation (Intel Xeon 32nm Core i7) Applies baseline feature set of Intel Xeon 32nm Corei7 (Westmere) processors to all hosts in the cluster. Compared to the Intel Xeon Corei7 mode, this EVC mode exposes additional CPU features including AES and PCLMULQDQ.

Note: Intel i3/i5 Xeon Clarkdale Series processors that do not support AESNI and PCLMULQDQ cannot be admitted to EVC modes higher than the Intel Xeon Corei7 mode.
Note: Intel Atom™ C2300-C2700 processors support the Intel "Westmere" Gen. EVC baseline although their architecture is different from the architecture of the Intel "Westmere" Generation processors.
Intel "Sandy Bridge" Generation Applies baseline feature set of Intel “Sandy Bridge” Generation processors to all hosts in the cluster.

This EVC mode exposes additional CPU features including AVX, XSAVE and ULE.

Note: Some Sandy Bridge micro architecture processors do not provide the full Sandy Bridge feature set. Such processors do not support EVC mode; they will only be admitted to the Intel Nehalem Generation mode or below.
Intel "Ivy Bridge" Generation Applies baseline feature set of Intel “Ivy Bridge” Generation processors to all hosts in the cluster.

This EVC mode exposes additional CPU features including ENFSTRING, F16C, FSGSBASE, SMEP and CPUID Faulting.
Intel "Haswell" Generation
Applies the baseline feature set of Intel "Haswell" Generation processors to all hosts in the cluster.

This EVC mode exposes additional CPU features including Advanced Vector Extensions 2, fused multiply-adds, Transactional Synchronization Extensions, and new bit manipulation instructions.
Intel "Broadwell" Generation

Applies the baseline feature set of Intel "Broadwell" Generation processors to all hosts in the cluster.

This EVC mode exposes additional CPU features including ADCX/ADOX, RDSEED, SMAP (CLAC/STAC), PREFETCHW,     RTM (Restricted Transactional Memory), and HLE (Hardware Lock Elision).
Intel "Skylake" Generation Applies the baseline feature set of Intel "Skylake" Generation processors to all hosts in the cluster.

This EVC mode exposes additional CPU features including Advanced Vector Extensions 512, Persistent Memory Support Instructions, Protection Key Rights, Save Processor Extended States with Compaction, and Save Processor Extended States Supervisor
Intel "Cascade Lake" Generation Applies the baseline feature set of Intel "Cascade Lake" Generation processors to all hosts in the cluster.
This EVC mode exposes additional CPU features including VNNI and XGETBV with ECX = 1.
Intel "Ice Lake" Generation Applies the baseline feature set of Intel "Ice Lake" Generation processors
to all hosts in the cluster.
This EVC mode exposes additional CPU features including SHA extensions,
Vectorized AES, User Mode Instruction Prevention, Read Processor ID, Fast Short
REP MOV, WBNOINVD, Galois Field New Instructions, and AVX512 Integer Fused
Multiply Add, Vectorized Bit Manipulation, and Bit Algorithms Instructions
Intel "Sapphire Rapids" Generation Applies the baseline feature set of Intel® "Sapphire Rapids" Generation processors (including Intel® "Emerald Rapids" Generation processors) to all hosts in the cluster. This EVC mode exposes additional CPU features, including Control-Flow Enforcement Technology, Advanced Matrix Extensions, Supervisor Protection Keys, AVX-VNNI, AVX512 FP16, AVX512 BF16, CLDEMOTE, SERIALIZE, WBNOINVD, and MOVDIRI instructions.

 

How can I find the EVC mode supported by a host?

In the vSphere Client connected to vCenter Server, the host Summary tab indicates whether EVC is enabled, and displays the current EVC mode for the host. Click the blue icon next to the EVC mode to display a list of all the supported EVC modes for the host.

If I have an EVC-enabled cluster set to a particular EVC mode, and I have a host outside the cluster that matches the EVC mode, can I migrate a virtual machine with vMotion between that host and the cluster?

Yes, but not always.  In some cases, the EVC mode does not include all of the features of a stand alone host.  When a stand alone host has more features than its corresponding EVC mode, then a VM that was powered-on on the stand alone host may not be able to be vmotioned to the EVC cluster because the cluster may be missing some features.  The missing features will be reported by the UI or in the vpxd log file.
If the VM was powered-on in the EVC cluster, then it can be vmotioned to the stand alone host, and back.
An EVC mode for a CPU generation may not have all of features of the hardware for various reasons, such as having to pick a common set of features from a set of server SKUs for a CPU generation.

How do I handle Intel Xeon E3 and E5 Processors or Intel Xeon X-series, E-series, L-series Processors that have different revision or series numbers (e.g. v2, v3, v4 or x5680, x5687, x5690)

When enabling EVC for a cluster with Intel processors of the Xeon E3 or Xeon E5 family that have different revision numbers (v2, v3, v4), an EVC baseline is required. This is due to new instruction sets being available in the different revisions. Although the processors will be the same EVC baseline, this is required for the processors to all present the same instruction sets to the virtual machines.

As with Xeon process with different revisions, when enabling EVC for a cluster with Intel processors of the same Xeon family (e.g. E56xx, X56xx, L56xx), an EVC baseline is required. This is due to new instruction sets being available in the different revisions. Although the processors will be the same EVC baseline, this is required for the processors to all present the same instruction sets to the virtual machines.

What is the difference between EVC and the old CPUID masking feature (accessed from the Virtual Machine Settings dialog box, Options tab, CPUID mask option)?

The older masking feature involved applying manual masks to individual virtual machines. EVC takes effect on a whole cluster and all virtual machines in the cluster. More accurately, EVC affects the hosts themselves, making all the hosts in the cluster appear to be the same type of CPU hardware, even if they are different.

Why is the . vmx config file for the virtual machine updated with CPUID bits?

The .vmx file contains information about the features that the virtual machine is using. vCenter Server need this information to provide accurate vMotion compatibility checks. These values are unrelated to the old CPUID mask values that also appear in the .vmx file.

What happens when a host is removed from an EVC-enabled cluster?

When a host leaves an EVC-enabled cluster, it reverts to its normal behavior. New virtual machines started on that host can access all the features of the CPU, and are not limited by the EVC mode that was in effect while the host was in the EVC cluster. Note that virtual machines that were once able to migrate to the host might no longer be permitted to do so.

If the EVC mode is raised, will virtual machines be able to access the new features available in that mode?

Not until the VM is powered off and then powered on. A warm reboot of the guest operating system is not sufficient to update the VMs EVC mode. A virtual machine determines which features are available to it at power on, and cannot access any new features in higher EVC modes until the VM is cold powered cycled (powered off, then powered on).

What is an ill-behaved application, and why does it affect EVC?

An ill-behaved application is one that does not use CPU-vendor-recommended methods of detecting features supported on a CPU. The recommended method is to run the CPUID instruction and look for the correct feature bits for the capabilities the application is expected to use. Unsupported methods used by ill-behaved applications include try-catch-fail or inferring the features present from the CPU version information. When unsupported methods are used, an application might detect features on a host in an EVC cluster that are being masked from the virtual machines. The CPUID-masking MSRs provided by CPU vendors do not disable the actual features. Therefore, an application can still use masked features. If a virtual machine running such an application is then migrated with vMotion to a host that does not physically support those features, the application might fail. VMware is not aware of any commercially-available ill-behaved applications. 

Can I create an EVC-enabled cluster if some of the hosts I want to add do not have AMD-V Extended Migration or Intel VT FlexMigration?

Yes. EVC-enabled clusters can contain hosts that do not feature AMD-V Extended Migration or Intel VT FlexMigration technology. For example, Intel Xeon Core2 processors do not have Intel VT FlexMigration technology, but can be added to clusters configured with an Intel Xeon Core2 EVC mode.

If I add newer hardware into an EVC-enabled cluster with a lower EVC mode, do I lose performance?

All CPU features provided by the host hardware are available to the hypervisor. Optimizations for CPU virtualization such as AMD-V and Intel VT-x or facilities for MMU virtualization such as AMD RVI or Intel EPT support are still used by the hypervisor. Only those CPU instructions that are unique to the new CPU are hidden from virtual machines when the host joins the EVC-enabled cluster. Typically this includes new SIMD instructions, such as the latest SSE additions. It is possible, but unlikely, that an application running in a virtual machine would benefit from these features, and that the application performance would be lower as the result of using an EVC mode that does not include the features. Check with the application vendor to determine which CPU features are used by the application.

How do I know when I can raise the EVC mode for a cluster?

The EVC mode can be raised for an EVC-enabled cluster if/when all hosts in the EVC cluster support the higher mode. This may occur periodically over time as older hosts are retired from the cluster and newer hosts are added. Use the Change EVC Mode dialog box to determine the EVC modes currently available to the hosts in a cluster.

Can I enable EVC on an existing non-EVC enabled cluster with powered-on virtual machines?
 
Yes. EVC can be enabled on an existing cluster. For more information, Enable EVC on an Existing Cluster

What is the impact on vSphere Cluster Features when I disable the EVC mode?

When disabling EVC mode on a cluster, it affects vSphere Cluster Features in the following ways:
  • vSphere HA (High Availability): vSphere HA is not impacted due to the failover power-cycling the virtual machines when starting them on a a new host. This allows the virtual machine to pick up the new CPU ID and allows it to start without a problem
  • vSphere DRS (Distributed Resource Scheduler):
    • vMotion:
      • Prior to power cycling the virtual machines:
        • They will maintain CPU level until a power cycle occurs.
        • DRS and vMotion operations will be the same as prior to disabling EVC mode.
      • Once the virtual machine is power cycled:
        • They are only able to move to other ESX/ESXi hosts that are at the same CPU generation or newer.
        • They will take on the CPU settings from the ESXi host it is running on. This can prevent DRS operations and vMotioning of the virtual machine to another host due to CPU incompatibility.
    • Storage vMotion: Virtual machines are able to be moved to Storage vMotion with EVC mode disabled within a cluster
  • Swapfile Location: This is not impacted.

Can I vMotion in an EVC disabled cluster with mixed ESXi host versions and mixed same family CPU architectures? 
 
No, an error will be received reporting that the host is incompatible. EVC must be enabled on a cluster for a VM to migrate between different CPU generations.

Additional Information

Powered by Wolken Software