Error: "One or more exceptions have occurred - Could not find object with moref "resgroup-#######"" while trying to deploy a new VM from tenant portal

Products

• VMware Cloud Director

Issue/Introduction

• When attempting to deploy a VM via Tenant Portal > Content Hub > vApp Templates, selecting any available vApp template results in the following error:

[########-####-####-####-############] One or more exceptions have occurred - Could not find object with moref "resgroup-#######" and VC ID "########-####-####-####-############" in inventory category "ResourcePool".

• On the Cloud Director(VCD) cell within the log file /opt/vmware/vcloud-director/logs/vcloud-container-debug.log the below error is present. 
  
  ERROR    | task-service-activity-pool-#| FutureUtil                     | Task future is not complete. | requestId=########-####-####-####-############,request=POST https://vcloud.example.com/api/vdc/<vcd-id>/action/instantiat...,requestTime=1775566292234,remoteAddress=<ipaddress>,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) Gecko/201...,accept=application/*+xml;version 39.0 vcd=########-####-####-####-############,task=########-####-####-####-############ activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:########-####-####-####-############)
com.vmware.ssdc.library.exceptions.InventoryObjectNotFoundException: Could not find object with moref "resgroup-####" and VC ID "########-####-####-####-############" in inventory category "ResourcePool".
        at com.vmware.vcloud.val.internal.impl.VC20VirtualEngine.checkInventoryRecord(VC20VirtualEngine.java:1243)
        at com.vmware.vcloud.val.internal.impl.VC20VirtualEngine.getDatastoresForResourcePool(VC20VirtualEngine.java:3213)
        at com.vmware.vcloud.fabric.storage.storedVm.impl.AbstractCreateStoredVmActivity.findDatastoreOnVc(AbstractCreateStoredVmActivity.java:899)
        at com.vmware.vcloud.fabric.storage.storedVm.impl.AbstractCreateStoredVmActivity.copyVmInVc(AbstractCreateStoredVmActivity.java:336)

Environment

VMware Cloud Director 10.6.x

Cause

This issue occurs due to a mismatch between the VMware vCenter inventory and the VMware Cloud Director database. Specifically, the error indicates that a resource pool referenced by VCD no longer exists or is out of sync with the vCenter Server.

Resolution

To resolve this issue, confirm that the resource pool exists in the vCenter Server. Once confirmed that it does exist, perform a reconnect of vCenter Server from Cloud Director in order to synchronize the managed object reference state.  

1. Inspect the error message from Cloud Director UI and record the 'resgroup' number mentioned in the error. Example: resgroup-12345
2. Connect to the vCenter Server MOB in a web browser and confirm that the resgroup managed object exists. Insert the resgroup-12345 as the moid reference.
   
   Example URL format: https://vcenter.example.com/mob/moid=resgroup-12345
   
   

   After accessing the vCenter MOB entry for resgroup-12345, verify the result:

   • If the page appears blank, the resource pool does not exist in the vCenter inventory.
   • If managed object information is displayed, the resource pool exists and is accessible.
3. If the resgroup does exist in vCenter Server, perform the steps outlined in Reconnect a vCenter Instance to Your VMware Cloud Director.
   
   The reconnect operation initiates a comprehensive scan of the vCenter Managed Object References (MoRefs). VCD queries vCenter for all Hosts, Clusters, Resource Pools, Networks, and VMs to ensure that the uuid and moref mapped in the VCD database match the current state in vCenter.
   
   
4. Attempt to deploy a VM in the Tenant to confirm it now works successfully.

If the resource pool does not exist, perform the following remediation steps:

1. Create a new resource pool in vCenter Server by following the steps in the official documentation: Create a resource pool

2. Execute the cell management tool to resolve the synchronization issue: Detect and Fix Resource Pool Mismatches between VMware Cloud Director and vCenter

Additional Information

Error: "Resource pool inventory is missing entry with moref" after disabling DRS on a Cloud Director Managed ESXi Cluster