"The requested resource was not found" error when failing over a VM in Cloud Director Availability 4.x

Products

• VMware Cloud Director

Issue/Introduction

• When performing a failover to a particular Organization VDC, the task fails and you see a similar error:

The requested resource was not found

• In the /opt/vmware/h4/cloud/log/cloud.log file on the Cloud Replication Management Appliance, you see entries similar to:

2022-07-06 08:27:31.884 ERROR - [UI-00000000-6a81-4b53-b23d-481e4bf00000-0000-Qr] [job-36] com.vmware.h4.jobengine.JobExecution     : Task 067ce6c3-172a-453e-97d9-bcb38b4b4390 (WorkflowInfo{type='failover', resourceType='vmReplication', resourceId='C4-01cf928f-0000-0000-b428-3dc700000000', isPrivate=false, resourceName='sourcevm'}) has failed
com.vmware.h4.exceptions.common.NotFoundException: The requested resource was not found

• In the /opt/vmware/h4/manager/log/manager.log file on the Cloud Replication Management Appliance, you see entries similar to:

2022-07-06 08:37:28.296 ERROR - [UI-00000000-6a81-4b53-b23d-481e4b00000-00000-XC-cU] [https-jsse-nio-8044-exec-9] c.v.h.c.c.error.ExceptionAdvisorBase     : A POST request from xxxx@org[127.0.0.1] to /inventory/datastore-accessibility failed. 

com.vmware.h4.exceptions.common.NotFoundException: The requested resource was not found

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue can occur because of a mismatch in the Cloud Director database where the Cloud Director resource pool records don't match the vCenter inventory's resource pool records.

Resolution

To validate if you are experiencing this issue, perform the following steps to check for a mismatch between the Cloud Director and vCenter inventory resource pool records.

1. SSH to the primary Cloud Director cell and log in as root.
2. Connect to the Cloud Director database:

su - postgres
psql vcloud

3. Run the following queries to check the Cloud Director and vCenter inventory resource pool records:

SELECT sub_rp_moref FROM vrp_rp WHERE vrp_id= (SELECT id FROM vrp WHERE name LIKE '%ORG_VDC_NAME%');
SELECT moref FROM resource_pool_inv WHERE name LIKE '%ORG_VDC_NAME%';

4. Compare the outputs from the two queries to confirm if there is a mismatch between them.
    

To resolve this issue contact Broadcom Support for Cloud Director and note this Article ID (315007) in the problem description. For more information, see Creating and managing Broadcom support cases