Custom resources and actions fail to execute after deleting and re-adding the Orchestrator integration.

Products

VMware Aria Suite

Issue/Introduction

Symptoms:

An Orchestrator instance added to vRealize Automation has been deleted and re-added.
Resource actions previously added from this Orchestrator instance fail with:

"<ResourceActionName> — Failed
There was a problem while building the execution context for the workflow run Caused by : Retries exhausted: 6/6"

Attempts to open and view these resource actions in the UI results in a "400 BAD_REQUEST" error:

400 BAD_REQUEST "Cannot retrieve form definition for runnable item RunnableItem(id=BD80808080808080808080808080808075C280800122528313869552e41805bb1, name=Shut down guest OS and wait, description=Shuts down a guest operating system and waits for the process to complete., projectId=null, type=vro.workflow, inputParameters=[RunnableItemParameter(type=VC:VirtualMachine, name=vm, description=Virtual machine to shut down)], outputParameters=null, endpointLink=/resources/endpoints/c25cf2bc-e7f9-4a99-a378-8a3a13446c21, inputBindings=null). Error message: 404 NOT_FOUND "vRealize Orchestrator Gateway received error 404 NOT_FOUND for request: http://provisioning-service.prelude.svc.cluster.local:8282/resources/endpoints/c25cf2bc-e7f9-4a99-a378-8a3a13446c21 error: {"message":"Service not found: http://provisioning-service.prelude.svc.cluster.local:8282/resources/endpoints/c25cf2bc-e7f9-4a99-a378-8a3a13446c21","statusCode":404,"errorCode":-2147483646,"serverErrorId":"2ad763f9-3fd9-4703-97f8-58590a3e4401","documentKind":"com:vmware:xenon:common:ServiceErrorResponse"}""

Attempts to alter existing custom resource types fails with a similar error:

400 BAD_REQUEST "Cannot retrieve form definition for runnable item RunnableItem(id=96f90f63-5668-441f-bea4-93b599dc3bcd, name=Copy of Create virtual machine folder, description=Creates a new virtual machine folder., projectId=null, type=vro.workflow, inputParameters=[RunnableItemParameter(type=VC:VmFolder, name=parentFolder, description=Parent folder), RunnableItemParameter(type=string, name=name, description=Name of the new folder)], outputParameters=[RunnableItemParameter(type=VC:VmFolder, name=newFolder, description=The newly created folder, if no errors occur)], endpointLink=/resources/endpoints/c25cf2bc-e7f9-4a99-a378-8a3a13446c21, inputBindings=null). Error message: 404 NOT_FOUND "vRealize Orchestrator Gateway received error 404 NOT_FOUND for request: http://provisioning-service.prelude.svc.cluster.local:8282/resources/endpoints/c25cf2bc-e7f9-4a99-a378-8a3a13446c21 error: {"message":"Service not found: http://provisioning-service.prelude.svc.cluster.local:8282/resources/endpoints/c25cf2bc-e7f9-4a99-a378-8a3a13446c21","statusCode":404,"errorCode":-2147483646,"serverErrorId":"24315c34-5e77-4ce3-9050-b62c005b6903","documentKind":"com:vmware:xenon:common:ServiceErrorResponse"}""

Environment

VMware vRealize Automation 8.x
VMware Aria Automation 8.x

Cause

If the same vRealize Orchestrator integration is removed and re-added in vRealize Automation the Custom Resources and Resource Actions will no longer be functional as they are linked to the original vRealize Orchestrator integration ID.

Resolution

For resource actions, these can be manually removed and re-added in the Aria Automation UI, this will resolve the conflict with the vRO integration id.

To resolve the issue with existing custom resource types that are referenced in custom forms the integration id can be updated to the new id using the forms service API:

1. Document the new vRO integration ID:

Open browser tools Network console and navigate to the newly re-added vRO integration. Cloud Assembly -> Infrastructure -> Integrations

The new Orchestrator integration id will be visible in the browsers call to the /iaas/api/integrations request url, see example below from a lab environment:

Request URL: https://<vRA-FQDN>/iaas/api/integrations/dcd46018-9712-4fe5-b825-59f58bae2e49?apiVersion=2021-07-15

The vRO integration ID also visible in the url of the browser if you navigate to the integration in the UI. The id is the last string after 'edit%2F'

2. Generate a bearer token to authenticate via API by following the steps in KB 89129

3. Query the forms service API to return a list of custom resources referenced in custom forms:

Issue a GET to url https://<vRA-FQDN>/form-service/api/custom/resource-types/ to return all the custom resource types referenced in custom forms

The vRO integration ID is referenced in the "endpointLink" property. In the screenshot below we see the stale integration id of c25cf2bc-e7f9-4a99-a378-8a3a13446c21

4. Document the id of any custom resource types which do not reference the new vRO integration id in their endpointLink properties.

5. Query the affected custom resource types individually by their ID.

Issue a GET to url https://<vRA-FQDN>/form-service/api/custom/resource-types/<ids from step 4>

6. Update the custom resource objects via the forms service API to reference the new Orchestrator integration ID:

For any affected custom resource objects leverage the body returned by the GET request in step 5 and alter the endpoint links to reference the new vRO integration Id documented step 1 then issue a POST to the same forms service API url with the corrected body content:

Update each affected custom resource object individually , you cannot update multiple records in a single POST.

https://<vRA-FQDN>/form-service/api/custom/resource-types/