Orchestrator data collection fails after updating Aria Automation certificates or redeploying external Orchestrator

Products

VMware Aria Suite

Issue/Introduction

Embedded Orchestrator is in a disconnected state under Assembler > Infrastructure > Integrations after updating the Aria Automation certificates.
The external Orchestrator had an issue and was re-deployed using the same FQDN
The UI contains the error:
910001: Can not connect to Automation Orchestrator Server, org.springframework.web.reactive.function.client.WebClientRequestException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Environment

VMware Aria Automation 8.x
VMware Aria Automation Orchestrator 8.x

Cause

The issue can have a number of causes:

Stale details cached in the browser.
SSL termination is configured on the load balancer. This causes the tool that manages the certificate to fail to find the new certificate when communicating to the VIP address, as the NLB configuration is configured with an untrusted certificate.
Old certificate information stored on the Orchestrator integration on the backend.

Resolution

Follow the resolution steps listed:

Note: before proceeding best practice dictates you first snapshot the Aria Automation appliances

1. Browser cache issue

Open a private browser window, navigate to Assembler > Infrastructure > Integrations and select VALIDATE. Accept the certificate if prompted If the issue persist proceed with validations below

2. Highly Available Architecture

In HA deployments, Configuring High Availability, validate that SSL passthrough is configured on the load-balancer. If SSL termination is enabled proceed with steps:

Remove the certificate from the network load balancer virtual IP interface to allow for a SSL passthrough configuration.
Run the following command to restart services.
```
/opt/scripts/deploy.sh
```
Once the services initialize navigate to Assembler > Infrastructure > Integrations and select VALIDATE

3. Validate the certificate stored for the Orchestrator integration via API

A. Obtain the Integration ID from the UI by opening the Orchestrator integration within the Aria Automation UI under Assembler > Infrastructure > Integrations

The Integration id portion of the url is everything after '%2F'

The integration id can also be validated in the web browser's developers tools:

B. Obtain a Bearer token to authenticate with the api following the article Generate Access_Token/Bearer_Token in VRA 8.x for API Authentication

C. To validate the current stored certificate submit a GET request to url:
https://<AriaAutomation_FQDN>/iaas/api/integrations/<vRO Integration ID>?apiVersion=2021-07-15

D. To view the certificate in a user friendly format copy it to a file, save it as a certificate extension .cer and open in windows.

E. If the old certificate is still stored you can update it via a Patch call. To obtain the new certificate in the correct format you can use the below command:

openssl s_client -connect <AriaAutomation_FQDN>:443 2> /dev/null | openssl x509 | awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}'

F. To update the certificate submit a PATCH call to url:

https://<AriaAutomation_FQDN>/iaas/api/integrations/<vROP Integration ID>?apiVersion=2021-07-15

with body:

{
"integrationProperties": {
"certificate": "valid certificate string in one line captured in step E"
}
}

A 202 accepted status indicates a successful request submission.

4. (Optional) As a last resort you can remove and re-add the Orchestrator integration

Note: Impacts of deleting the integration:

Resource actions are disassociated
Orchestrator content sources are disassociated
To repair existing custom resource objects you can follow the steps contained in Custom resources and actions fail to execute after deleting and re-adding the Orchestrator integration

Within Resource actions and Orchestrator content sources, remove and re-import the workflows to re-associate.