Converting VMEM & VMSS Files From an Encrypted VM

Products

• VMware vSphere ESXi

Issue/Introduction

Converting a VMware virtual machine snapshot to a memory dump with vmss2core.exe fails with the following error :

Unrecognized .vmss file (magic 4f434e45)

 

Example:

#vmss2core.exe -W8 snapshot.vmsn snapshot.vmem
vmss2core version 8456865 Copyright (C) 1998-2017 VMware, Inc. All rights reserved.
Unrecognized .vmss file (magic 4f434e45).

Environment

ESXi 8.x

Cause

If the VM is encrypted, the .vmem file is encrypted as well, and the conversion operation will fail.

Resolution

As an alternative to snapshot, suspend the VM, creating .vmem  and .vmss files.

Decrypt .vmem and vmss files in ESXi:

crypto-util encobj decrypt --related vm-name.vmx  vm-name-######.vmem vm-name-######-decrypted.vmem

crypto-util encobj decrypt --related vm-name.vmx  vm-name-######.vmss vm-name-######-decrypted.vmss

 

Additional Information

If a snapshot must be used, follow the resolution steps in KB:  Unable to convert a VMware virtual machine snapshot to a memory dump