Embedded Orchestrator Integration is broken after replacing certificate.

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Embedded VRO integration is not in healthy state after Aria Automation certificate replacement

910001: Can not connect to Automation Orchestrator Server. org.springframework.web.reactive.function.client.WebClientRequestException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Environment

VMware Aria Automation 8.x
VMware Aria Automation Orchestrator 8.x

Cause

When the Orchestrator certificate is replaced, trust is broken between Aria Automation and Aria Automation Orchestrator.

Resolution

To workaround this issue you need to update the Integration endpoint data using API
Steps for Aria Automation without multi tenancy:

1. Obtain a Bearer token following the article Generate an Access Token and Bearer Token in VMware Aria Automation for API Authentication
2. Configure shell variables for retrieved token and hostname:
```
TOKEN="<Token>"
VRA=$(vracli load-balancer)
```
3. Get the certificate chain of the Orchestrator endpoint
```
openssl s_client -connect $VRA:443 -showcerts < /dev/null | sed -ne '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p'| awk 'NF {sub(/\r/, "");  printf "%s\\n",$0;}'
```
4. Get the info and ID of the embedded Orchestrator Integration:
```
curl -k "https://$VRA/iaas/api/integrations/?apiVersion=2021-07-15" -H "Authorization: Bearer $TOKEN" | jq -r '.content[] | .id + "," + .name'
```
  Example: results:
  <Integration-ID>,embedded-ABX-onprem
  <Integration-ID>,raas
  <Integration-ID>,gss support
  <Integration-ID>,embedded-VRO
  
  Note: The <Integration-ID> can be retrieved from Assembler > Infrastructure > Connections > Integrations > embedded-VRO in the browser address bar as seen below, id taken after %2F
5. Review the Orchestrator Integration details:
  
  curl -k "https://$VRA/iaas/api/integrations/<Integration ID captured in step-4>?apiVersion=2021-07-15" -H "Authorization: Bearer $TOKEN" | jq .
6. Patch the integration using API:
  
  curl -v -k -X PATCH "https://$VRA/iaas/api/integrations/<Integration ID captured in step-4>?apiVersion=2021-07-15" -H "Accept: application/json" -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" --data-raw '{"integrationProperties": {"certificate":"Certificate content captured in step 3"}, "customProperties": {"certificate":"Certificate content captured in step 3"}}'
  
  Body with just certificate:
```
{
   "integrationProperties": {
      "certificate": "Certificate content captured in step 3"
      },
   "customProperties ": {
      "certificate": "Certificate content captured in step 3"
      }
 }
```
7. After applying the certificate, check the Integration properties as in Step 5.
8. After the certificate has been applied, the data collection should recover, however in certain circumstances the same error may still be seen and require a recovery of either property:
  
  set empty for "vroUnresponsiveReason":""
  or set "vro-responsive": "true"
  1. Patch the integration using API:
    
    curl -v -k -X PATCH "https://$VRA/iaas/api/integrations/<Integration ID captured in step-3>?apiVersion=2021-07-15" -H "Accept: application/json" -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" --data-raw '{"customProperties": {"vroUnresponsiveReason":""}}'
    
    Body with just vroUnresponsiveReason:
```
{
  "customProperties":  
  {
    "vroUnresponsiveReason":""
  }
}
```
9. If the update integration fails due to properties to be read-only, then please contact Broadcom support