Manual or Backup Software Snapshots fail with error: Unable to create snapshot, Invalid virtual machine configuration

Products

• VMware vSphere ESXi

Issue/Introduction

• VM snapshot from the vSphere client fails with error message:
  
  Status: Invalid virtual machine configuration
  
  
• Snapshots by Backup Software fail with error:
  
  Unable to create snapshot, Unable create non-quiesced snapshot after a retry attempt: Invalid virtual machine configuration
  
  
• Delete snapshot task fails with:
  
  A general system error occurred: Fault cause: vim.fault.GenericVmConfigFault
  
  
• The affected VM's vmware.log file shows the below error:

2020-08-14T12:02:41.602Z| vmx| I125: SNAPSHOT: SnapshotConfigInfoReadEx: Unable to load dict from '/vmfs/volumes/vsan/##/Test-VM.vmx'.
2020-08-14T12:02:41.602Z| vmx| I125: SNAPSHOT: SnapshotConfigInfoReadEx failed for file '/vmfs/volumes/vsan/##/Test-VM.vmx': Dictionary problem (6)

• Prior to the snapshot failures, a storage connectivity issue was noticed in the logs

2020-08-14T08:27:42.813Z: [vmfsCorrelator] 13214789879183us: [vob.vmfs.heartbeat.timedout] 
2020-08-14T08:27:42.813Z: [vmfsCorrelator] 13214762211408us: [esx.problem.vmfs.heartbeat.timedout]

2020-08-14T08:31:59.347Z: [vmfsCorrelator] 13215046414757us: [vob.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume

Environment

VMware vSphere ESXi 6.7

Cause

The snapshot dictionary file - <VM Name>.vmsd.usd - gets updated for every snapshot and is stored in two places.

1. In memory in a string buffer and
2. On the File-System in the VM folder. The in-memory file gets over-written when the host or VM restarts.

The dictionary file could get corrupt on either of these places. The storage outage seems to be the trigger for this corruption and is likely that the in-memory and on-disk file metadata were out-of-sync, causing the snapshot failures.

Resolution

There is no permanent resolution for this issue at this time. 

Workaround

Workaround Options

1. Rename the vmsd file and once a new snapshot is taken, a new vmsd file will be created. To perform this workaround, follow the below steps:
   1. Make a note of the datastore where the VM is stored.
   2. Connect to the ESXi host with an SSH session.
   3. Navigate to the datastore

cd /vmfs/volumes/DATASTORE/VM_Name/
      
      
   4. List all the files in the folder:

ls -l
      
      
   5. Rename the .vmsd file to .vmsd.old

mv VM_Name.vmsd VM_Name.vmsd.old


   6. Take or Remove the Snapshot (according to the initial issue)
      
      
2. vMotion the affected VM to another host and test the snapshot. vMotion creates a new VM world and would use a new in-memory dictionary file.
   
   
3. Power cycle the VM (OFF->ON)
   
   
4. If the above workarounds didn't help, the only option is to Power OFF the VM, unregister from vCenter, re-register and power on the VM.