Embedded Orchestrator Integration shows disconnected after replacing the certificate

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Embedded VRO integration is not in healthy state after Aria Automation certificate replacement

910001: Can not connect to Automation Orchestrator Server. org.springframework.web.reactive.function.client.WebClientRequestException:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target;

The provisioning-service-app Logs report the below events:

provisioning [host='provisioning-service-app-<PoD ID>' thread='ebs-1' user='' org='' trace='' parent='' span=''] c.v.p.e.EventBrokerSubscriptionManager.lambda$publishEvent$17:436 - [ebs-integration] Published event Event[id='<Event ID>'(endpoint.cud), orgId='<Org ID>', correlation='<Reference ID>'(contextId), userName='N/A'], Data= {endpointType=vro, customProperties={hostName=https://<Orchestrator FQDN>:443, commonName=<Automation FQDN>, validSince=1755739184000, vroUnresponsiveReason=910001: Can not connect to Automation Orchestrator Server. org.springframework.web.reactive.function.client.WebClientRequestException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed, endpointHealthCheckState=AVAILABLE

The tango-vro-gateway-app logs reports the below events:

202Y-MM-DDT09:HH:MM.121Z ERROR tango-vro-gateway [host='tango-vro-gateway-app-<PoD ID>' thread='reactor-http-epoll-4' user='' org='' trace='' parent='' span=''] c.v.a.v.g.e.e.OnPremVroHealthStatusChecker.lambda$checkVroHealth$0:88 - Error while getting list of endpoints to check their health status:

  org.springframework.web.reactive.function.client.WebClientRequestException: finishConnect(..) failed: Connection refused: provisioning-service.prelude.svc.cluster.local/##.##.##.##:8282

 at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:136) ~[spring-webflux-6.0.19.jar:6.0.19]

Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:

Error has been observed at the following site(s):

 *__checkpoint ⇢ Request to GET http://provisioning-service.prelude.svc.cluster.local:8282/provisioning/mgmt/endpoints [DefaultWebClient]

...

 Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: provisioning-service.prelude.svc.cluster.local/##.##.##.##:8282

Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused

Environment

VMware Aria Automation 8.x
VMware Aria Automation Orchestrator 8.x

Cause

When the Orchestrator certificate is replaced, trust is broken between Aria Automation and Aria Automation Orchestrator.

Resolution

To workaround this issue you need to update the Integration endpoint data using API

Steps for Aria Automation::

1. Obtain a Bearer token following the article Generate an Access Token and Bearer Token in VMware Aria Automation for API Authentication
  Note: Get the Bearer token for the local Administrator account that was configured during the installation of Automation. In many cases this will be configadmin.
2. Configure shell variables for retrieved token and hostname:
```
TOKEN="<Token>"
VRA=$(vracli load-balancer)
```
3. Get the certificate chain of the Orchestrator endpoint
```
openssl s_client -connect $VRA:443 -showcerts < /dev/null | sed -ne '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p'| awk 'NF {sub(/\r/, "");  printf "%s\\n",$0;}'
```
4. Get the info and ID of the embedded Orchestrator Integration:
```
curl -k "https://$VRA/iaas/api/integrations/?apiVersion=2021-07-15" -H "Authorization: Bearer $TOKEN" | jq -r '.content[] | .id + "," + .name'
```
  Example: results:

<Integration-ID>,embedded-ABX-onprem
<Integration-ID>,raas
<Integration-ID>,gss support
<Integration-ID>,embedded-VRO

Note: The <Integration-ID> can be retrieved from Assembler > Infrastructure > Connections > Integrations > embedded-VRO in the browser address bar as seen below, id taken after %2F

5. Review the Orchestrator Integration details:

curl -k "https://$VRA/iaas/api/integrations/<Integration ID captured in step-4>?apiVersion=2021-07-15" -H "Authorization: Bearer $TOKEN" | jq .

6. Patch the integration using API:

curl -v -k -X PATCH "https://$VRA/iaas/api/integrations/<Integration ID captured in step-4>?apiVersion=2021-07-15" -H "Accept: application/json" -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" --data-raw '{"integrationProperties": {"certificate":"Certificate content captured in step 3"}, "customProperties": {"certificate":"Certificate content captured in step 3"}}'

Body with just certificate:

{
"integrationProperties": {
"certificate": "Certificate content captured in step 3"
},
"customProperties ": {
"certificate": "Certificate content captured in step 3"
}
}

7.After applying the certificate, check the Integration properties as in Step 5.

8. After the certificate has been applied, the data collection should recover, however in certain circumstances the same error may still be seen and require a recovery of either property:

set empty for "vroUnresponsiveReason":""
or set "vro-responsive": "true"

1. 1. Patch the integration using API:
    
    curl -v -k -X PATCH "https://$VRA/iaas/api/integrations/<Integration ID captured in step-3>?apiVersion=2021-07-15" -H "Accept: application/json" -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" --data-raw '{"customProperties": {"vroUnresponsiveReason":""}}'
    
    Body with just vroUnresponsiveReason:

{
"customProperties":
{
"vroUnresponsiveReason":""
}
}

9.If the update integration fails due to properties to be read-only, then please contact Broadcom support