Migration to a different processor

Products

• VMware
• VMware Desktop Hypervisor
• VMware vSphere ESXi

Issue/Introduction

In vSphere, Virtual Machines (VMs) can be migrated from one host to another to move workloads between compute resources. There are two ways to migrate VMs: Live migration, and Cold migration.

• Live Migration (vMotion) - A non-disruptive transfer of a virtual machine from one host to another. For a Live Migration, the source host and target host must provide the same CPU functions (CPU flags). Live Migration requires the source and destination hosts to have CPUs from the same manufacturer, and only CPU functions which both hosts support are provided (same CPU generation, or by using VMware Enhanced vMotion Compatibility (EVC)). 

• Cold Migration - The virtual machine on the original host is first shut down, then started on the new host. When the VM starts on the new host, the VM goes through an Operating System initialization where the Operating System can adapt to utilize all available CPU functions. In cold migration, VMs can be migrated across servers with different CPUs, including different manufacturers, or CPU generations, in most cases. 

In general, Operating Systems and applications are designed to be compatible with a range of popular CPUs. A “well behaved” Operating System/application determines the underlying CPU and uses what is provided. This behavior allows cold migration to work across different CPUs. However, in some cases, users can optimize an Operating System/application at compile/install time for a certain CPU. If an Operating System/application is compiled/installed for a certain CPU, it might not work on the destination host after a cold migration. This behavior is attributed to the different NUMA topologies presented on different CPUs. 

As an example, during installation, many distributions of Linux choose a kernel that is optimized for the specific processor on which it is being installed. Other distributions install a generic kernel by default, but provide architecture-specific kernels that the user can choose to install. The kernel might contain instructions that are available only on that processor. These instructions can have adverse effects on a host with the wrong type of processor. Thus, A Linux virtual machine created on a host might not work when you migrate it to a host with a significantly different processor, e.g. from a host with Intel processor to a host with AMD processor, a host with AMD processor to a host with Intel processor, or even between hosts with different generations of processors from the same vendor if EVC is not used. 
This problem is not specific to virtual machines and can occur on physical computers. For example, if you move a hard drive with a Linux installation from an AMD machine to an Intel machine, you might experience problems trying to boot from that drive.

Resolution

Confirm a generic kernel is used by the VM before you migrate a virtual machine between hosts when they are running processors that are significantly different, from a different CPU vendor, or different CPU generations (and not using EVC). If a kernel that is optimized for a specific processor is used for a VM, don't migrate the virtual machine to a host with a significantly different processor.