Powering on a virtual machine from a suspend state or reverting to a snapshot fails

Products

VMware vSphere ESXi

Issue/Introduction

Provide information on how to troubleshoot and fix VM which will not power on due to suspend or snapshot state problems.

Symptoms:

Powering on a virtual machine (VM) from a suspend state, or reverting the VM to a snapshot with memory state saved, fails with an error.

For example:

Cannot use host <hostname> because the host's CPU is not compatible with the virtual machine's suspend state.
Failed to revert the execution state of the virtual machine <vm name> on host <host name>
Module 'CheckpointLate' power on failed
The features supported by the processors in this machine are different from the features supported by the processors in the machine on which the checkpoint was saved. Try to resume the snapshot on a machine where the processors have the same features.
The virtual CPU performance counter events are not compatible with the host CPU.

In the vmware.log file, you see similar to:

[YYYY-MM-DDTHH:MM:SS]| vmx| I124: [msg.checkpoint.restore.error] An error caused the restore operation to fail. Cancel the restore operation and correct the error, or discard the snapshot's state and power off. The saved snapshot will not be affected.

Environment

VMware vSphere ESXi 6.0
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Cause

This issue occurs when the virtual machine is suspended on an ESXi host with one CPU type, and then moved to a host with a different CPU type. The machine selected for the virtual machine needs to support the exact same CPU features as the machine on which the snapshot or suspend was performed.

You may see these symptoms when moving suspended VMs or reverting on hosts with:

The exact same CPU but with different BIOS or ESXi version builds. This is because these patches can change or mask some CPU features to resolve bugs or security vulnerabilities.
The exact same CPU but with different BIOS features enabled in the setup configuration
Host clusters with differing Enhanced vMotion Compatibility (EVC) settings, including clusters with EVC on to clusters with it off, or the reverse.

Resolution

How to prevent CPU conflict by fixing VM on original ESXi host

If the previous ESXi host and the current host for the VM have compatible hardware, BIOS version, and ESXi OS version, make sure that identical hardware features are enabled on the the BIOS of the source host and the new host

Note: If you are migrating to a new cluster with Enhanced vMotion Compatibility (EVC) enabled, the EVC level will determine which CPU features are available. See Enhanced vMotion Compatibility (EVC) Explained

To resolve this issue bring the virtual machine back to the host on which it was suspended to power it on or to revert to snapshot.
If the VM power on still fails, connect to the ESXi host client directly and re-try the power on.
- If this succeeds, there is likely a communication problem between the ESXi host and vCenter and was not related to incompatible CPU features.
If the virtual machine needs to be moved to an incompatible machine, use power off instead of suspend. Make sure that the snapshots do not contain virtual machine memory saved, or that they are powered off during snapshot.
If you are unable to migrate the virtual machine from host one to another due to hardware compatibility issues, unregister the virtual machine from the host and re-register on original host. See How to register or add a Virtual Machine (VM) to the vSphere Inventory in vCenter Server

How to power on VM on host without bringing it back to the original host

Caution: These steps cause all unsaved data to be lost.

A. If this is a suspended VM:

- If the new ESXi host has a more advanced CPU feature set than original host, the VM may present an exclamation mark on the VM and a message asking you to answer the question very similar to the Snapshot example images below. Answer to use the Discard option as for the Snapshot example below.
- If the new ESXi host has a less advanced feature set, the UI will fail to power on the VM with an error. You must then follow these steps:

1. 1. Connect to the host through SSH or console where the suspended virtual machine resides
  2. Log in as a user with administrator privileges.
  3. Run this command to change directory to the folder of the suspended virtual machine:
    
    cd /vmfs/volumes/<Datastore name>/<virtual machine name>
  4. Open the .vmx file in a text editor. See Editing files on an ESXi host using vi
    1. Remove the .vswp reference under sched.swap.derivedName.
    2. Remove the .vmss reference under checkpoint.vmState.
    3. Save the changes and exit the text editor.
  5. Run this command to get the virtual machine world ID:
    
    vim-cmd vmsvc/getallvms
  6. Run this command to reload virtual machine .vmx file:
    
    vim-cmd vmsvc/reload <vmid from previous step>
    
    Within a minute the virtual machine's status in vCenter Server shows as powered off instead of suspended. You can then power up the machine as normal.

B. If this problem is for reverting a snapshot with memory:

1. The VM object will show an exclamation mark symbol.
2. Go to the Summary tab and select the Answer Question link as per the image below

Note: If the question does not appear, try powering on the VM again to trigger it.

In the Answer Question dialog, select the Discard option, then press the OK button. as seen in the image below:

Within a couple of minutes the VM will show as powered off.
Try powering it on.
If the power on still fails, you will need to follow the steps in How to prevent CPU conflict by fixing VM on original host

Additional Information

Starting with ESXi 7.0 Update 3l, you can take a memory snapshot of a virtual machine with an independent disk only to analyze the guest operating system behavior of a virtual machine. You cannot use such snapshots for virtual machine backups because restoring this type of snapshots is unsupported.
Refer Snapshot Limitations

Impact/Risks:
You cannot power on the suspended virtual machine or revert to a snapshot.