Cannot create new snapshots on a virtual machine after removing a disk from the virtual machine

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

If you have a virtual machine with multiple disks that is running a snapshot, you may experience these symptoms if you remove a disk from the virtual machine and manually delete the VMDK for the removed disk:

Creating a snapshot of a virtual machine fails.
You see the error:

Invalid virtual machine configuration
In ESXi 5.0, you may also see the error:

an invalid snapshot configuration was detected
After restarting the management agents, the virtual machine goes into an invalid state.
When you re-register the virtual machine by removing it from the inventory and then reading it, the virtual machine appears as invalid.
The *.vmsd file has no relevant data and its size is reported as 1 byte or else it is corrupted.
In the vmware.log file, you see entries similar to:

<YYYY-MM-DD> <time>.872: vmx| SNAPSHOT: SnapshotConfigInfoRead: Unable to load dict from '/vmfs/volumes/6a2956d6-########/xyz/xyz.vmsd'.
May 27 05:21:02.873: vmx| SNAPSHOT: SnapshotConfigInfoRead failed for file ' xyz .vmx': Dictionary problem (6)
</time>
In the /var/log/hostd.log file, you see entries similar to:

[<YYYY-MM-DD> <time>.405 F5AD9B90 info 'vm:/vmfs/volumes/6a2956d6-########/ xyz / xyz .vmx'] Reloading config state.
[<YYYY-MM-DD> <time>.426 F5AD9B90 info 'Libs'] SNAPSHOT: SnapshotConfigInfoRead: Unable to load dict from '/vmfs/volumes/6a2956d6-########/xyz/xyz.vmsd'.
[<YYYY-MM-DD> <time>.426 F5AD9B90 info 'Libs'] SNAPSHOT: SnapshotConfigInfoRead failed for file '/vmfs/volumes/6a2956d6-########/xyz/xyz.vmx': Dictionary problem (6)
[<YYYY-MM-DD> <time>.426 F5AD9B90 info 'Libs'] Snapshot_GetTree(/vmfs/volumes/6a2956d6-########/ xyz / xyz .vmx) failed: Dictionary problem.
....
[<YYYY-MM-DD> <time>.426 F5AD9B90 info 'Libs'] VMHSSnapshotPublishTree failed: Dictionary problem
[<YYYY-MM-DD> <time>.426 F5AD9B90 info 'Libs'] VMHSSnapshotLoad failed: Dictionary problem
[<YYYY-MM-DD> <time>.426 F5AD9B90 info 'Libs'] VMHSVMLoadConfig: Load failed but vmdb tree modified to reflect offline state
[<YYYY-MM-DD> <time>.426 F5AD9B90 info 'Libs'] VMHSVMLoadConfig failed: Dictionary problem (Message)
[<YYYY-MM-DD> <time>.426 F5AD9B90 info 'Libs'] VMHS_ManageVM failed: Dictionary problem
[<YYYY-MM-DD> <time>.427 F63BC900 info 'Vmsvc'] Foundry_CreateEx failed: Error: (1) Unknown error
[<YYYY-MM-DD> <time>.427 F63BC900 info 'vm:/vmfs/volumes/6a2956d6-########/ xyz / xyz .vmx'] Failed to load virtual machine.
[<YYYY-MM-DD> <time>.427 F63BC900 info 'vm:/vmfs/volumes/6a2956d6-########/ xyz / xyz .vmx'] Failed to load virtual machine. Marking as una vailable: vmodl.fault.SystemError
[<YYYY-MM-DD> <time>.427 F63BC900 info 'vm:/vmfs/volumes/6a2956d6-########/ xyz / xyz .vmx'] Marking VirtualMachine invalid
[<YYYY-MM-DD> <time>.427 F63BC900 info 'vm:/vmfs/volumes/6a2956d6-########/ xyz / xyz .vmx'] State Transition (VM_STATE_INITIALIZING -> VM_STATE_INVALID_CONFIG)
[<YYYY-MM-DD> <time>.427 F63BC900 verbose 'vm:/vmfs/volumes/6a2956d6-########/ xyz / xyz .vmx'] Time to load virtual machine: 137 (msecs)

<YYYY-MM-DD>T<time>.250Z [61F91B70 info 'vm:SNAPSHOT: SnapshotConfigInfoReadEx failed for file '/vmfs/volumes/4e82430d-########-####-0025b5000005/xyz/xyz.vmx'] : Dictionary problem (6)
<YYYY-MM-DD>T<time>.252Z [61F91B70 info 'vm:Snapshot_GetTree(/vmfs/volumes/4e82430d-########-####-0025b5000005/xyz/xyz.vmx'] failed: Dictionary problem.
<YYYY-MM-DD>T<time>.252Z [61F91B70 info 'Libs'] VMHSSnapshotPublishTree failed: Dictionary problem
<YYYY-MM-DD>T<time>.252Z [61F91B70 info 'Libs'] VMHSSnapshotLoad failed: Dictionary problem
<YYYY-MM-DD>T<time>.252Z [61F91B70 info 'Libs'] VMHSVMLoadConfig failed: Dictionary problem (Message)
<YYYY-MM-DD>T<time>.252Z [61F91B70 info 'Libs'] VMHSVMLoadConfig retried set ops successfully to reflect offline state
<YYYY-MM-DD>T<time>.252Z [61F91B70 info 'Libs'] VMHS_ManageVMVigor failed: Dictionary problem
<YYYY-MM-DD>T<time>.252Z [611C2B70 info 'Vmsvc.vm:/vmfs/volumes/4e82430d-########-####-0025b5000005/xyz/xyz.vmx'] Foundry_[Create|Open]Ex failed: Error: (1) Unknown error
<YYYY-MM-DD>T<time>.252Z [611C2B70 info 'Vmsvc.vm:/vmfs/volumes/4e82430d-########-####-0025b5000005/xyz/xyz.vmx'] Failed to load virtual machine.
<YYYY-MM-DD>T<time>.252Z [611C2B70 warning 'Vmsvc.vm:/vmfs/volumes/4e82430d-########-####-0025b5000005/xyz/xyz.vmx'] Failed to load VM from foundry during register vmodl.fault.SystemError
<YYYY-MM-DD>T<time>.252Z [611C2B70 info 'Vmsvc.vm:/vmfs/volumes/4e82430d-########-####-0025b5000005/xyz/xyz.vmx'] Marking VirtualMachine invalid
<YYYY-MM-DD>T<time>.252Z [611C2B70 info 'Vmsvc.vm:/vmfs/volumes/4e82430d-########-####-0025b5000005/xyz/xyz.vmx'] State Transition (VM_STATE_INITIALIZING -> VM_STATE_INVALID_CONFIG)</time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time>

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0

Cause

This issue occurs if the removed disk is still in the snapshot database. If you examine the vmware.log file for the virtual machine, it contains an entry that indicates that it cannot find the base VMDK for the disk that was removed. This may occur if the files were manually deleted via the service console or the datastore browser, instead of deleting through the Edit Settings dialog of the virtual machine or if the .vmsd file is corrupted as well.

Resolution

To resolve this issue, consolidate the snapshots of the virtual machine.

To consolidate the snapshots of the virtual machine:

Create a back up of the virtual machine.
In VMware Infrastructure (VI) Client, browse the datastore where the virtual machine is located.
Verify that there is enough free space on the datastore to commit the snapshots.
Right-click on the file with the .vmsd extension and rename it from vmname.vmsd to vmname.vmsd.old.
From VI Client, create a new snapshot.
Click Delete all in the Snapshot Manager to consolidate the snapshots. This commits the newest snapshot, plus the existing snapshots.
Delete all of the .vmsd files from the virtual machine directory.
Delete all the .vmsn files from the virtual machine directory.

Note: These files are old memory snapshots which are no longer usable.
Restart the management agents. For more information, see Restarting the Management agents on an ESX or ESXi Server (1003490).
After you have enough disk space on the datastore where the virtual machine is stored, create a snapshot of the virtual machine.