Manual or Backup Software Snapshots fail with error: Unable to create snapshot, Invalid virtual machine configuration

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual machine snapshot attempt from the vSphere client fails with error message:
  
  Status: Invalid virtual machine configuration
  
  
• Snapshots by Backup Software may fail with error similar to below:
  
  Unable to create snapshot, Unable create non-quiesced snapshot after a retry attempt: Invalid virtual machine configuration
  
  
• Delete snapshot task fails with error similar to below:
  
  A general system error occurred: Fault cause: vim.fault.GenericVmConfigFault
  
  
• The affected VM's vmware.log file shows entries similar to below:

YYYY-MM-DDThh:mm:ss vmx| I125: SNAPSHOT: SnapshotConfigInfoReadEx: Unable to load dict from '/vmfs/volumes/vsan/##/Test-VM.vmx'.
YYYY-MM-DDThh:mm:ss vmx| I125: SNAPSHOT: SnapshotConfigInfoReadEx failed for file '/vmfs/volumes/vsan/##/Test-VM.vmx': Dictionary problem (6)

• Prior to the snapshot failures, storage connectivity issues might be noticed on the vobd.log on the ESXi host wrt the datastore where the virtual machine resides

YYYY-MM-DDThh:mm:ss [vmfsCorrelator] ###: [vob.vmfs.heartbeat.timedout] 
YYYY-MM-DDThh:mm:ss [vmfsCorrelator] ###: [esx.problem.vmfs.heartbeat.timedout]

YYYY-MM-DDThh:mm:ss [vmfsCorrelator] ###: [vob.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume ###

Environment

VMware vSphere ESXi 7.x, 8.x

Cause

The snapshot dictionary file "<vm_name>.vmsd" gets updated for every snapshot and the data is stored in two places.

1. In memory in a string buffer on the ESXi host
2. On the File-System in the VM folder. 

The in-memory file gets over-written when the host or VM restarts.

The dictionary file data could get corrupt on either of these places , with possible cause being any storage connectivity triggering this corruption as the in-memory and on-disk file metadata goes out-of-sync, thus causing the snapshot failures.

Resolution

To address the issue, follow the below steps to recreate the virtual machine snapshot dictionary file:

1. Make a note of the datastore where the virtual machine is stored.
2. Connect to the ESXi host where the virtual machine currently resides with an SSH session.
3. Navigate to the datastore

cd /vmfs/volumes/<DATASTORE>/<vm_name>/
   
   
4. List all the files in the folder:

ls -l
   
   
5. Rename the <vm_name>.vmsd file to <vm_name>.vmsd.old

mv VM_Name.vmsd VM_Name.vmsd.old


6. Take a new snapshot and delete it.

In case the issue still persists, reload the virtual machine config (vmx) file .For further information, please refer to below articles:

Reloading a vmx file without removing the virtual machine from inventory

An error occurred while taking a snapshot: Dictionary problem.