Unable to convert a Windows 11 VM snapshot to a memory dump

Products

VMware vSphere ESXi

Issue/Introduction

Converting a VMware virtual machine snapshot to a memory dump with vmss2core.exe fails with the below error due to the disks being encrypted.

Unrecognized .vmss file (magic 4f434e45)

Example:

Environment

ESXi 8.x

Cause

This issue occurs because a windows 11 VM is encrypted by default due to the mandatory Trusted Platform Module (TPM) requirement for the operating system.

Resolution

Decryption of vmem and vmss files in ESXi can be performed as below: (Navigate to the VM's folder)

crypto-util encobj decrypt --related vm-231.vmx vm-231.vmem vm-231-decrypted.vmem

crypto-util encobj decrypt --related vm-231.vmx vm-231.vmss vm-231-decrypted.vmss

2. Once decryption is completed

3.Transfer the .vmss and .vmem (If present) file to the system where vmss2core is located.

4.Run vmss2core on the Copied Files

./vmss2core -W8 vm-231-decrypted.vmss vm-231-decrypted.vmem

Additional Information

Following the Decrypt an Encrypted Virtual Machine or Virtual Disk is not an option as a windows 11 VM is vTPM-enabled and to remove a vTPM from a virtual machine, see Remove Virtual Trusted Platform Module from a Virtual Machine.

Data loss warning appears : Removing the vTPM on an encrypted virtual machine will cause you to lose any data protected by it, such as BitLocker encryption keys. You must back up and decrypt any data before proceeding, or the data will be unrecoverable.

Also removing the vTPM will not let the windows 11 guest OS to boot.

Note: For VMs with other guest operating systems you can follow Unable to convert a VMware virtual machine snapshot to a memory dump

If you run vmss2core tool and get an output including this message, it means the files do not contain usable data:
- Error parsing Windows data.
  Cannot create memory.dmp
The OS needs to have loaded the kernel in order to get usable data for vmss2core. Examples of VMs in a boot state that would not have usable data:
- You see a message on boot that says:
  Windows failed to start. A recent hardware or software change might be the cause.
- The VM boot up does not progress past the BIOS boot screen
Examples of a VM that would have usable data:
- The VM booted with a Windows Blue Screen error (BSOD)
- The VM booted and a Linux kernel panic occurred
- The VM is frozen after the OS boots

For additional information, see the Debugging Virtual Machines with the Checkpoint to Core Tool Guide.

Suspending a virtual machine on ESX/ESXi to collect diagnostic information

Note : For the vmss2core tool to successfully process snapshot files from a Windows virtual machine, features such as Virtualization-Based Security (VBS) and Hyper-V must be disabled. If either of these features is enabled, the tool will instead produce a .core memory dump rather than a standard format.