Error: "One or more exceptions have occurred - Could not find object with moref "resgroup-#######"" while trying to deploy a new VM from tenant portal

Products

• VMware Cloud Director

Issue/Introduction

• When attempting to deploy a VM via Tenant Portal > Content Hub > vApp Templates, selecting any available vApp template results in the following error:

[########-####-####-####-############] One or more exceptions have occurred - Could not find object with moref "resgroup-#######" and VC ID "########-####-####-####-############" in inventory category "ResourcePool".

• On the Cloud Director(VCD) cell within the log file /opt/vmware/vcloud-director/logs/vcloud-container-debug.log the below error is present. 
  
  ERROR    | task-service-activity-pool-#| FutureUtil                     | Task future is not complete. | requestId=########-####-####-####-############,request=POST https://vcloud.example.com/api/vdc/<vcd-id>/action/instantiat...,requestTime=1775566292234,remoteAddress=<ipaddress>,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) Gecko/201...,accept=application/*+xml;version 39.0 vcd=########-####-####-####-############,task=########-####-####-####-############ activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:########-####-####-####-############)
com.vmware.ssdc.library.exceptions.InventoryObjectNotFoundException: Could not find object with moref "resgroup-####" and VC ID "########-####-####-####-############" in inventory category "ResourcePool".
        at com.vmware.vcloud.val.internal.impl.VC20VirtualEngine.checkInventoryRecord(VC20VirtualEngine.java:1243)
        at com.vmware.vcloud.val.internal.impl.VC20VirtualEngine.getDatastoresForResourcePool(VC20VirtualEngine.java:3213)
        at com.vmware.vcloud.fabric.storage.storedVm.impl.AbstractCreateStoredVmActivity.findDatastoreOnVc(AbstractCreateStoredVmActivity.java:899)
        at com.vmware.vcloud.fabric.storage.storedVm.impl.AbstractCreateStoredVmActivity.copyVmInVc(AbstractCreateStoredVmActivity.java:336)

Environment

VMware Cloud Director 10.6.x

Cause

This issue occurs due to a mismatch between the VMware vCenter inventory and the VMware Cloud Director database. Specifically, the error indicates that a resource pool referenced by VCD no longer exists or is out of sync with the vCenter Server.

Resolution

To resolve this issue firstly confirm that the resource pool exists in the vCenter Server. Once confirmed that it does exist, perform a reconnect of vCenter Server from within Cloud Director in order to synchronize the managed object reference state.  

1. Inspect the error message from Cloud Director UI and record the 'resgroup' number mentioned in the error. Example: resgroup-12345
2. Connect to the vCenter Server MOB in a web browser and confirm that the resgroup managed object exists. Insert the resgroup-12345 as the moid reference.
   
   Example URL format: https://vcenter.example.com/mob/moid=resgroup-12345
   
   Once connected to the vCenter MOB entry for resgroup-12345, If you see a blank white page, then the resource pool does not exist. If you see actual managed object data displayed, the resource pool does exist.
   
   Note: If the resgroup does not exist, refer to the Additional Information section below for guidance.
   
   
3. If the resgroup does exist in vCenter Server, perform the steps outlined in Reconnect a vCenter Instance to Your VMware Cloud Director.
   
   The reconnect operation initiates a comprehensive scan of the vCenter Managed Object References (MoRefs). VCD queries vCenter for all Hosts, Clusters, Resource Pools, Networks, and VMs to ensure that the uuid and moref mapped in the VCD database match the current state in vCenter.
   
   
4. Attempt to deploy a VM in the Tenant to confirm it now works successfully.

Additional Information

Detect and Fix Resource Pool Mismatches between VMware Cloud Director and vCenter
Error: "Resource pool inventory is missing entry with moref" after disabling DRS on a Cloud Director Managed ESXi Cluster
テナントポータルから仮想マシンデプロイを試みると "One or more exceptions have occurred - Could not find object with moref "resgroup-#######"" エラーが発生する