Unable to delete a Cloud Director Availability failed over virtual machine from VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• After performing a clean-up of a test failover in Cloud Director Availability, the test virtual machine in Cloud Director cannot be deleted.
• Manually deleting the virtual machine from the Cloud Director Portal fails, and you see a message similar to:

  Could not find object with moref "vm-####" and VC ID "########-####-####-####-############" in inventory category "VirtualMachine".

• Or you may encounter following error while trying to remove the vApp:

  The entity Ref: com.vmware.vcloud.entity.vm:<VM-UUID> is busy completing an operation VAPP_VM_RELOAD_FROM_VC. VAPP_VM_RELOAD_FROM_VC(com.vmware.vcloud.entity.task:######-####-#########-#########)

• In the vSphere Client, you see that the test virtual machine was successfully deleted and is no longer present in the vCenter Server inventory.

Environment

• VMware Cloud Director 10.x
• VMware Cloud Director Availability 4.x

Cause

This issue can occur when there is a communication issue between Cloud Director and vCenter Server while the test failover clean-up task is ongoing, resulting in the virtual machine being removed on the vCenter level and the Cloud Director records being stranded.

Resolution

This will need the modification in the VMware Cloud Director DB, so please make sure to take the  backup of the VMware Cloud Director Database: Backup and Restore of VMware Cloud Director Appliance
Please open a case with Broadcom Support to do the modification in VCD Database. For more information, see Creating and managing Broadcom support cases

Additional Information

You may encounter similar issue if you are using any 3rd party backup/DR solution.