Deleting virtual machine or snapshot fails with error: vim.fault.GenericVmConfigFault

Products

VMware vSphere ESXi

Issue/Introduction

Snapshot consolidation for a virtual machine with multiple snapshots might fail with a "vim.fault.GenericVmConfigFault"

Deleting a virtual machine from disk that has multiple snapshots may also fail with the same error mentioned above using CLI.

[root@hostname:~] vim-cmd vmsvc/snapshot.removeall 3
Remove All Snapshots:
Remove all snapshots failed: (vmodl.fault.SystemError) {
   faultCause = (vmodl.MethodFault) null,
   faultMessage = <unset>,
   reason = "Fault cause: vim.fault.GenericVmConfigFault"
   msg = "A general system error occurred: Fault cause: vim.fault.GenericVmConfigFault.

The var/run/log/hostd.log file reports "An error occurred while deleting a snapshot: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot".

YYYY-MM-DDTHH:MM:SSZ verbose hostd[21##69] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:################-################/########-####-####-####-############/VM_NAME.vmx opID=m6###xyt-417##903-auto-ou5qo-h5:739##492-f1-12-772b user=vpxuser:####\#######] Removeallsnapshots received. Consolidate: true
YYYY-MM-DDTHH:MM:SSZ info hostd[2104769] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:################-################/########-####-####-####-############/VM_NAME.vmx opID=m6##6xyt-417###03-auto-ou5qo-h5:7####492-f1-12-772b user=vpxuser:####\#######] State Transition (VM_STATE_ON -> VM_STATE_REMOVEALL_SNAPSHOT)
YYYY-MM-DDTHH:MM:SSZ verbose hostd[210##2] [Originator@6876 sub=Vigor.Vmsvc.vm:/vsan:################-################/########-####-####-####-############/VM_NAME.vmx opID=m####xyt-41##3861-auto-on7pi-h5:7###4520-ad-01-01-01-01-6a-59ae] Delete snapshot 2##6 and its children message: An error occurred while deleting a snapshot: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot.
-->
YYYY-MM-DDTHH:MM:SSZ warning hostd[21##69] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:################-################/########-####-####-####-############/VM_NAME.vmx opID=m####6xyt-41####03-auto-ou5qo-h5:7####492-f1-12-772b user=vpxuser:####\#######] Failed to remove snapshot(s): N3Vim5Fault20GenericVmConfigFault9ExceptionE(Fault cause: vim.fault.GenericVmConfigFault
--> )

The /vmfs/volumes/datastore_UUID/VM_NAME/vmware.log file reports the following error.

YYYY-MM-DDTHH:MM:SSZ In(05) vmx m6836xyt-41###903-auto-ou5qo-h5:7##25492-f1-12-772b SNAPSHOT: SnapshotDiskTreeAddFromSnapshot: Trying to add snapshot VM_NAME-Snapshot2##7.vmsn to
 disk /vmfs/volumes/vsan:################-################/########-####-####-####-############/VM_NAME.vmdk which already has snapshot VM_NAME-Snapshot2##6.vmsn.
YYYY-MM-DDTHH:MM:SSZ In(05) vmx m6836xyt-41###903-auto-ou5qo-h5:7##25492-f1-12-772b SNAPSHOT: SnapshotGenerateDeleteDisks Failed to fetch disk tree: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)
YYYY-MM-DDTHH:MM:SSZ In(05) vmx m6836xyt-4###7903-auto-ou5qo-h5:7###5492-f1-12-772b SNAPSHOT: SnapshotDeleteNode failed: One of the disks in this virtual machine is already in use by avirtual machine or by a snapshot (21)
YYYY-MM-DDTHH:MM:SSZ In(05) vmx m6836xyt-41717903-auto-ou5qo-h5:7###5492-f1-12-772b SNAPSHOT: Snapshot_Delete failed: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)

Example of Incorrect .vmsd File Hierarchy:

less /vmfs/volumes/<datastore_uuid>/VM_Name/vm_name.vmsd

.encoding = "###-#"
snapshot.lastUID = "7"
snapshot.current = "7"
snapshot0.uid = "5"
snapshot0.filename = "vm_name-Snapshot5.vmsn"
snapshot0.displayName = "VM Snapshot DATE, TIME"
snapshot0.createTimeHigh = "408330"
snapshot0.createTimeLow = "-206508375"
snapshot0.numDisks = "3"
snapshot0.disk0.fileName = "vm_name.vmdk"
snapshot0.disk0.node = "scsi0:0"
snapshot0.disk1.fileName = "vm_name_1.vmdk"
snapshot0.disk1.node = "scsi0:1"
snapshot0.disk2.fileName = "vm_name_2.vmdk"
snapshot0.disk2.node = "scsi0:2"
snapshot.numSnapshots = "3"
snapshot1.uid = "6"
snapshot1.filename = "vm_name-Snapshot6.vmsn"
snapshot1.parent = "5"
snapshot1.displayName = "VM Snapshot DATE, TIME"
snapshot1.createTimeHigh = "408330"
snapshot1.createTimeLow = "-200076526"
snapshot1.numDisks = "3"
snapshot1.disk0.fileName = "vm_name.vmdk"      <<< snapshot1.disk0.fileName should point to vm_name-000001.vmdk instead of vm_name.vmdk. The same applies to the other disks. >>>
snapshot1.disk0.node = "scsi0:0"
snapshot1.disk1.fileName = "vm_name_1.vmdk"
snapshot1.disk1.node = "scsi0:1"
snapshot1.disk2.fileName = "vm_name_2.vmdk"
snapshot1.disk2.node = "scsi0:2"
snapshot2.uid = "7"
snapshot2.filename = "vm_name-Snapshot7.vmsn"
snapshot2.parent = "6"
snapshot2.displayName = "VM Snapshot DATE, TIME"
snapshot2.createTimeHigh = "408330"
snapshot2.createTimeLow = "-191878359"
snapshot2.numDisks = "3"
snapshot2.disk0.fileName = "vm_name-000001.vmdk"
snapshot2.disk0.node = "scsi0:0"
snapshot2.disk1.fileName = "vm_name_1-000001.vmdk"
snapshot2.disk1.node = "scsi0:1"
snapshot2.disk2.fileName = "vm_name_2-000001.vmdk"
snapshot2.disk2.node = "scsi0:2"

Environment

vSphere 7.0.x
vSphere 8.0.x

Cause

Corruption or invalid hierarchy in the .vmsd snapshot metadata file prevents vSphere from correctly fetching the disk tree.

Resolution

To solve this issue, please apply the following steps:

1. Using vSphere Client, shut down the guest operating system and power the virtual machine off.
2. Next, find the virtual machine folder where the .vmx configuration file of the VM is located.
  - open an SSH session to the host where the VM is currently registered.
  - the file /etc/vmware/hostd/vmInventory.xml lists all VMs the host currently holds with their configuration file path.
  - Alternatively use the command below to list all virtual machines on the host by using the esxcli command.
    
    # esxcli vm process list
  - or search for the specific .vmx file:
    
    # find /vmfs/volumes -iname <vm_name>.vmx
3. Please take note of path to the .vmx file and unregister the VM from the vCenter inventory:
  - Right-click the VM in vSphere Client
  - Select "Unregister from Inventory" (be careful not to select "Delete from Disk"):
4. Return back to the SSH session with the ESXi, and change into the VM folder noted in Step 2:
  
  # cd /vmfs/volumes/<datastore_name>/<<vm_name>>
5. Create a temp folder within this VM folder
  
  # mkdir vm_snapshot_temp
6. Move all files with the extensions .vmsn (the snapshot state files) and .vmsd (the snapshot directory file) and from the VM folder into this new folder.
  
  # mv *.vmsn ./vm_snapshot_temp/
  # mv *.vmsd ./vm_snapshot_temp/
7. Return to the vSphere Client and use the datastore browser to locate the .vmx file of the virtual machine.
8. Select the file and click on "Register VM" to reregister the virtual machine with the vCenter Server.
9. Acknowledge the ‘VM consolidation needed’ status and wait for the consolidation process to complete.
10. Once the consolidation process has finished, verify that there are no more snapshots in the virtual machine folder, then power the virtual machine back on

Note: After moving the VMSN and VMSD files, the virtual machine can be re-registered and select "Remove All Snapshots" to delete all existing snapshots.
If the intent is to delete the virtual machine entirely, re-register the VM and then choose "Delete from Disk" to remove it completely.

Additional Information

Delete Snapshot Task Fails With Error A general system error occurred: Fault cause: vim.fault.GenericVmConfigFault | Failed to fetch disk tree