Manual or Backup Software Snapshots fail with error: Unable to create snapshot, Invalid virtual machine configuration

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• VM snapshot from the vSphere client fails with error message: Status: Invalid virtual machine configuration
• Snapshots by Backup Software fail with error:  Unable to create snapshot, Unable create non-quiesced snapshot after a retry attempt: Invalid virtual machine configuration
• Cannot delete snapshot with error "A general system error occurred: Fault cause: vim.fault.GenericVmConfigFault"
• The affected VM's vmware.log file shows the below error:

2020-08-14T12:02:41.602Z| vmx| I125: SNAPSHOT: SnapshotConfigInfoReadEx: Unable to load dict from '/vmfs/volumes/vsan:52f1a618da63b0a7-9e0c60fb59ebea94/cc587d5b
-9430-61f1-3334-246e96b1eab8/Test-VM.vmx'.
2020-08-14T12:02:41.602Z| vmx| I125: SNAPSHOT: SnapshotConfigInfoReadEx failed for file '/vmfs/volumes/vsan:52f1a618da63b0a7-9e0c60fb59ebea94/cc587d5b-9430-61f1-3334-246e96b1eab8/Test-VM.vmx': Dictionary problem (6)

• Prior to the Snapshot failures, a Storage connectivity issue was noticed in the logs

2020-08-14T08:27:42.813Z: [vmfsCorrelator] 13214789879183us: [vob.vmfs.heartbeat.timedout] 5b7d58cc-d15d441b-e8f7-246e96b1eab8 cc587d5b-9430-61f1-3334-246e96b1eab8
2020-08-14T08:27:42.813Z: [vmfsCorrelator] 13214762211408us: [esx.problem.vmfs.heartbeat.timedout] 5b7d58cc-d15d441b-e8f7-246e96b1eab8 cc587d5b-9430-61f1-3334-246e96b1eab8

The VM Test-VM's folder (cc587d5b-9430-61f1-3334-246e96b1eab8 - on vSAN) was affected by this outage as well  --> 

2020-08-14T08:31:59.347Z: [vmfsCorrelator] 13215046414757us: [vob.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume 5b7d58cc-d15d441b-e8f7-246e96b1eab8  cc587d5b-9430-61f1-3334-246e96b1eab8): [Timeout] [HB state abcdef02 offset 3391488 gen 13 stampUS 13215046413517 uuid 5e6ca709-e890aca2-776b-246e96b1cb48 jrnl <FB 1755560> drv 14.81]

Environment

VMware vSphere ESXi 6.7

Cause

The snapshot dictionary file  - <VM Name>.vmsd.usd - gets updated for every snapshot and is stored in two places - (1) in memory in a string buffer and (2) on the File-System in the VM folder. The in-memory file gets over-written when the host or VM restarts.

The dictionary file could get corrupt on either of these places. The storage outage seems to be the trigger for this corruption and in this case, it was likely that the in-memory and on-disk file metadata were out-of-sync, causing the Snapshot Failures.

Resolution

1. There is no permanent resolution for this issue at this time. Please see the work-around section for potential solutions.

Workaround:

There are multiple options to work-around this issue (each is a separate workaround):

Note: Each workaround is a separate fix.

• Rename the vmsd file and once a new Snapshot is taken, a new vmsd file will be created

To perform this workaround , follow the below steps:

1. Make a note of the datastore where the VM is stored.
2. Connect to the ESXi host with an SSH session.
3. Navigate to the datastore from step 4.

cd /vmfs/volumes/DATASTORE/VM_Name/

4. List all the files in the folder with: ls -l
5. Rename the .vmsd file to .vmsd.old

mv VM_Name.vmsd VM_Name.vmsd.old

6. Take or Remove the Snapshot (according to the initial issue)

OR

•  vMotion the affected VM to another host and test the snapshot - vMotion creates a new VM world and would use a new in-memory dictionary file.

OR

• Power cycle the VM (OFF->ON)

OR

• If the above workarounds didn't help, the only option is to  Power OFF the VM, unregister from vCenter, Re-Register  and Power ON the VM. For more information about unregistering VM and register it back to ESXi Host, see Virtual machines appear to be running or registered on multiple ESX/ESXi Servers (1005051)