Custom resources and actions fail to execute after deleting and re-adding the Orchestrator integration

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Symptoms:

An Orchestrator instance added to Aria Automation has been deleted and re-added.
Resource actions previously added from this Orchestrator instance fail with:

"<ResourceActionName> — Failed
There was a problem while building the execution context for the workflow run Caused by : Retries exhausted: 6/6"

Attempts to open and view these resource actions in the UI results in a "400 BAD_REQUEST" error:

400 BAD_REQUEST "Cannot retrieve form definition for runnable item RunnableItem(id=#################################################################, name=Shut down guest OS and wait, description=Shuts down a guest operating system and waits for the process to complete., projectId=null, type=vro.workflow, inputParameters=[RunnableItemParameter(type=VC:VirtualMachine, name=vm, description=Virtual machine to shut down)], outputParameters=null, endpointLink=/resources/endpoints/########-#####-####-####-########, inputBindings=null). Error message: 404 NOT_FOUND "vRealize Orchestrator Gateway received error 404 NOT_FOUND for request: http://provisioning-service.prelude.svc.cluster.local:8282/resources/endpoints/########-#####-####-####-######## error: {"message":"Service not found: http://provisioning-service.prelude.svc.cluster.local:8282/resources/endpoints/########-#####-####-####-########","statusCode":404,"errorCode":-2147483646,"serverErrorId":"########-#####-####-####-########","documentKind":"com:vmware:xenon:common:ServiceErrorResponse"}""

Attempts to alter existing custom resource types fails with a similar error:

400 BAD_REQUEST "Cannot retrieve form definition for runnable item RunnableItem(id=########-#####-####-####-########, name=Copy of Create virtual machine folder, description=Creates a new virtual machine folder., projectId=null, type=vro.workflow, inputParameters=[RunnableItemParameter(type=VC:VmFolder, name=parentFolder, description=Parent folder), RunnableItemParameter(type=string, name=name, description=Name of the new folder)], outputParameters=[RunnableItemParameter(type=VC:VmFolder, name=newFolder, description=The newly created folder, if no errors occur)], endpointLink=/resources/endpoints/########-#####-####-####-########, inputBindings=null). Error message: 404 NOT_FOUND "vRealize Orchestrator Gateway received error 404 NOT_FOUND for request: http://provisioning-service.prelude.svc.cluster.local:8282/resources/endpoints/########-#####-####-####-######## error: {"message":"Service not found: http://provisioning-service.prelude.svc.cluster.local:8282/resources/endpoints/########-#####-####-####-########","statusCode":404,"errorCode":-2147483646,"serverErrorId":"########-#####-####-####-########","documentKind":"com:vmware:xenon:common:ServiceErrorResponse"}""

Environment

VMware Aria Automation 8.x

Cause

If the same Aria Orchestrator integration is removed and re-added in Aria Automation the Custom Resources and Resource Actions will no longer be functional as they are linked to the original Aria Orchestrator integration ID.

Resolution

For resource actions, these can be manually removed and re-added in the Aria Automation UI, this will resolve the conflict with the Aria Orchestrator integration id.

To resolve the issue with existing custom resource types that are referenced in custom forms the integration id can be updated to the new id using the forms service API:

1. Document the new Aria Orchestrator integration ID:

Open browser tools Network console and navigate to the newly re-added Aria Orchestrator integration. Cloud Assembly -> Infrastructure -> Integrations

The new Orchestrator integration id will be visible in the browsers call to the /iaas/api/integrations request url, see example below from a lab environment:

Request URL: https://<Aria-Automation-FQDN>/iaas/api/integrations/#######-####-####-#############?apiVersion=2021-07-15

The Aria Orchestrator integration ID also visible in the url of the browser if you navigate to the integration in the UI. The id is the last string after 'edit%2F'

2. Generate a bearer token to authenticate via API by following the steps in KB 89129

3. Query the forms service API to return a list of custom resources referenced in custom forms:

Issue a GET to url https://<Aria-Automation-FQDN>/form-service/api/custom/resource-types/ to return all the custom resource types referenced in custom forms

The Aria Orchestrator integration ID is referenced in the "endpointLink" property. In the screenshot below we see the stale integration id of ########-####-####-####-##########

4. Document the id of any custom resource types which do not reference the new Aria Orchestrator integration id in their endpointLink properties.

5. Query the affected custom resource types individually by their ID.

Issue a GET to url https://<Aria-Automation-FQDN>/form-service/api/custom/resource-types/<ids from step 4>

6. Update the custom resource objects via the forms service API to reference the new Orchestrator integration ID:

For any affected custom resource objects leverage the body returned by the GET request in step 5 and alter the endpoint links to reference the new Aria Orchestrator integration Id documented step 1 then issue a POST to the same forms service API url with the corrected body content:

Update each affected custom resource object individually , you cannot update multiple records in a single POST.

https://<Aria-Automation-FQDN>/form-service/api/custom/resource-types/