Error: "The object 'vim.vm.Snapshot:snapshot-####' has already been deleted or has not been completely created" while deleting a Snapshot

Products

• VMware Cloud Director

Issue/Introduction

• Attempting to delete a  snapshot from Cloud Director results in the following error:
  
  [ UUID] Internal Server Error - (vmodl.fault.ManagedObjectNotFound) { faultCause = null, faultMessage = null, obj = ManagedObjectReference: type = VirtualMachineSnapshot, value = snapshot-####, serverGuid = null } - The object 'vim.vm.Snapshot:snapshot-####' has already been deleted or has not been completely created
  
  
• From the VMware Cloud Director (VCD) UI the following Snapshot details are shown:

no name, type = external, timestamp = 31/12/1969 07:00, size = 0MB

or

no name, type = external, timestamp = 01/01/1970 01:00, size = 0MB

• Ghost snapshot on VM.
• Phantom Null Snapshot Issue.
• Following API Returns:

Request URL: https://[Cloud Director URL]/cloudapi/1.0.0/vm/urn:vcloud:vm:########-####-####-####-############/snapshots?page=1&pageSize=128&links=true

"values": [
 {
 "id": "########-####-####-####-############",
 "name": null,
 "isCurrent": false,
 "state": null,
 "storageConsumedMb": 0,
 "dateCreated": null,
 "memory": false,
 "quiesce": false,
 "vappSnapshotId": null,
 "parentSnapshotId": null,
 "vm": {
 "name": "[VM Name]",
 "id": "urn:vcloud:vm:########-####-####-####-############"
 },
 "type": "External"
 }
 ]

• VM has no snapshot at vCenter level.
• Veeam Plug-in for VMware Cloud Director is used by Tenants to perform backup operations.

Environment

VMware Cloud Director 10.6.x

Cause

This issue occurs when snapshot metadata remains in the VCD database after the snapshot is removed from vCenter. Common triggers include:

• Third Party creating and Deleting Snapshots quickly.
• Temporary issue with Cloud Director receiving updates from vCenter leading to Cloud Director missing the update that this Snapshot was deleted.
• Race condition in Cloud Director. With tasks from vCenter where the Snapshot is created and deleted within a rapid time. Leading to stale data.

Resolution

To resolve this issue, please open a Support Request with Broadcom Technical Support and reference this KB article 397884.

Database modifications may be required to resolve this issue and will require VMware Technical Support assistance.