Unable to convert a Windows 11 VM snapshot to a memory dump

Products

• VMware vSphere ESXi

Issue/Introduction

• Converting VMware virtual machine snapshots to a memory dump with vmss2core.exe fails with the error: "Unrecognized .vmss file (magic 4f434e45)." This article provides steps to resolve the issue for Windows 11 virtual machines, which are encrypted by default due to mandatory Trusted Platform Module (vTPM) requirements.

# ./vmss2core.exe -W8 snapshot.vmsn snapshot.vmem
vmss2core version 24832109 Copyright (C) 1998-2025 Broadcom. All Rights Reserved.
Unrecognized .vmss file (magic 4f434e45).

• Running vmss2core generated a 0 kb memory.dmp file

Environment

• vSphere ESXi 8.x
• Guest OS: Windows 11, Windows Server 2025

Cause

• Windows 11 virtual machines are encrypted by default due to vTPM requirements. The vmss2core tool cannot process encrypted .vmsn or .vmem files directly. Additionally, ensure the latest version of vmss2core is being used, as older versions do not support modern operating systems.

Resolution

1) Download the file vmss2core.exe and the file _zlib1.dll See KB Obtaining the latest version of vmss2core.exe without installing VMware Workstation Pro

Copy the file vmss2core.exe and the file _zlib1.dll to a windows workstation in the same folder

Rename file _zlib1.dll to zlib1.dll

 

2) Suspend the target VM to create the required .vmss and .vmem files. 

Navigate to the folder containing the VM files in ESXi and proceed in decrypting both the vmem and vmss files:     

crypto-util encobj decrypt --related vm-###.vmx  vm-###.vmem vm-###-decrypted.vmem
crypto-util encobj decrypt --related vm-###.vmx  vm-###.vmss vm-###-decrypted.vmss

 

3) Transfer the decrypted .vmss and .vmem files to the system where vmss2core has been downloaded

Run vmss2core against the decrypted .vmss and .vmem files:    

 ./vmss2core -W8 vm-###-decrypted.vmss vm-###-decrypted.vmem

 

 

Additional Information

• Suspending a virtual machine on ESX/ESXi to collect diagnostic information

  • For the vmss2core tool to successfully process snapshot files from a Windows virtual machine, features such as Virtualization-Based Security (VBS) and Hyper-V must be disabled.

  • If either of these features is enabled, the tool will instead produce a .core memory dump rather than a standard format.

• For older versions of Windows, see Unable to convert a VMware virtual machine snapshot to a memory dump

• Following the Decrypt an Encrypted Virtual Machine or Virtual Disk is not an option as a windows 11 VM is vTPM-enabled and to remove a vTPM from a virtual machine, see Remove Virtual Trusted Platform Module from a Virtual Machine.

Warning: data loss

⚠️ Removing TPM will render all encrypted data on this VM unrecoverable, are you sure you want to proceed?

• • You must back up and decrypt any data before proceeding, or the data will be unrecoverable.

• Removing the vTPM will will prohibit Windows 11 guest OS from booting. 

• If you run vmss2core tool and get an output including this message, it means the files do not contain usable data:

  • Error parsing Windows data.

    Cannot create memory.dmp

• The OS needs to have loaded the kernel in order to get usable data for vmss2core. Examples of VMs in a boot state that would not have usable data:

  • You see a message on boot that says:

    Windows failed to start. A recent hardware or software change might be the cause. 

  • The VM boot up does not progress past the BIOS boot screen

• Examples of a VM that would have usable data:

  • The VM booted with a Windows Blue Screen error (BSOD)

  • The VM booted and a Linux kernel panic occurred 

  • The VM is frozen after the OS boots