Error "The requested resource was not found" when failing over a VM in Cloud Director Availability

Products

• VMware Cloud Director

Issue/Introduction

• When performing a failover to a particular Organization VDC, the task fails and you see a similar error:

The requested resource was not found

• In the /opt/vmware/h4/cloud/log/cloud.log file on the Cloud Replication Management Appliance, you see entries similar to:

2022-07-06 08:27:31.884 ERROR - [UI-########-####-####-####-############-####-Qr] [job-36] com.vmware.h4.jobengine.JobExecution     : Task ######-####-####-####-########## (WorkflowInfo{type='failover', resourceType='vmReplication', resourceId='C4-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', isPrivate=false, resourceName='sourcevm'}) has failed
com.vmware.h4.exceptions.common.NotFoundException: The requested resource was not found

• In the /opt/vmware/h4/manager/log/manager.log file on the Cloud Replication Management Appliance, you see entries similar to:

2022-07-06 08:37:28.296 ERROR - [UI-########-####-####-####-############-####-cU] [https-jsse-nio-8044-exec-9] c.v.h.c.c.error.ExceptionAdvisorBase     : A POST request from xxxx@org[127.0.0.1] to /inventory/datastore-accessibility failed. 

com.vmware.h4.exceptions.common.NotFoundException: The requested resource was not found

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x

VMware Cloud Director 10.x

Cause

This issue can occur because of a mismatch in the Cloud Director database where the Cloud Director resource pool records don't match the vCenter inventory's resource pool records.

Resolution

To resolve this issue,

1. SSH to the primary Cloud Director cell and log in as root.
2. Connect to the Cloud Director database:
   • su - postgres
   • psql vcloud
3. To validate if you are experiencing this issue, perform the following steps to check for a mismatch between the Cloud Director and vCenter inventory resource pool records.
4. Run the following queries to check the Cloud Director and vCenter inventory resource pool records:
   • SELECT sub_rp_moref FROM vrp_rp WHERE vrp_id= (SELECT id FROM vrp WHERE name LIKE '%ORG_VDC_NAME%');
   • SELECT moref FROM resource_pool_inv WHERE name LIKE '%ORG_VDC_NAME%';
5. Compare the outputs from the two queries to confirm if there is a mismatch between them.
6. Run this command to fix the resource pool mismatch: /opt/vmware/vcloud-director/bin/cell-management-tool detect-rp-mismatches --fix-mismatch-with-vc

If issue still exists, contact Broadcom Support for Cloud Director and note this Article ID (315007) in the problem description. For more information, see Creating and managing Broadcom support cases

Additional Information

Detect and Fix Resource Pool Mismatches between VMware Cloud Director and vCenter