Embedded Orchestrator Integration shows disconnected after replacing the certificate.
search cancel

Embedded Orchestrator Integration shows disconnected after replacing the certificate.

book

Article ID: 380251

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

  • Embedded VRO integration is not in healthy state after Aria Automation certificate replacement

     

910001: Can not connect to Automation Orchestrator Server. org.springframework.web.reactive.function.client.WebClientRequestException:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target;

  • The provisioning-service-app Logs report the below events:

provisioning [host='provisioning-service-app-<PoD ID>' thread='ebs-1' user='' org='' trace='' parent='' span=''] c.v.p.e.EventBrokerSubscriptionManager.lambda$publishEvent$17:436 - [ebs-integration] Published event Event[id='<Event ID>'(endpoint.cud), orgId='<Org ID>', correlation='<Reference ID>'(contextId), userName='N/A'], Data= {endpointType=vro, customProperties={hostName=https://<Orchestrator FQDN>:443, commonName=<Automation FQDN>, validSince=1755739184000, vroUnresponsiveReason=910001: Can not connect to Automation Orchestrator Server. org.springframework.web.reactive.function.client.WebClientRequestException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed, endpointHealthCheckState=AVAILABLE

  • The tango-vro-gateway-app logs reports the below events:
202Y-MM-DDT09:HH:MM.121Z ERROR tango-vro-gateway [host='tango-vro-gateway-app-<PoD ID>' thread='reactor-http-epoll-4' user='' org='' trace='' parent='' span=''] c.v.a.v.g.e.e.OnPremVroHealthStatusChecker.lambda$checkVroHealth$0:88 - Error while getting list of endpoints to check their health status:
  org.springframework.web.reactive.function.client.WebClientRequestException: finishConnect(..) failed: Connection refused: provisioning-service.prelude.svc.cluster.local/##.##.##.##:8282
at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:136) ~[spring-webflux-6.0.19.jar:6.0.19]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
  Error has been observed at the following site(s):
*__checkpoint ⇢ Request to GET http://provisioning-service.prelude.svc.cluster.local:8282/provisioning/mgmt/endpoints [DefaultWebClient] 
...
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: provisioning-service.prelude.svc.cluster.local/##.##.##.##:8282
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused

 

Environment

  • VMware Aria Automation 8.x
    VMware Aria Automation Orchestrator 8.x

Cause

When the Orchestrator certificate is replaced, trust is broken between Aria Automation and Aria Automation Orchestrator.

Resolution

To workaround this issue you need to update the Integration endpoint data using API

Manual/API Method

Steps for Aria Automation::

    1. Obtain a Bearer token following the article Generate an Access Token and Bearer Token in VMware Aria Automation for API Authentication

    2. Configure shell variables for retrieved token and hostname:

      TOKEN="<Token>"
VRA=$(vracli load-balancer)


    3. Get the certificate chain of the Orchestrator endpoint

      openssl s_client -connect $VRA:443 -showcerts < /dev/null | sed -ne '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p'| awk 'NF {sub(/\r/, "");  printf "%s\\n",$0;}'


    4. Get the info and ID of the embedded Orchestrator Integration:
      curl -k "https://$VRA/iaas/api/integrations/?apiVersion=2021-07-15" -H "Authorization: Bearer $TOKEN" | jq -r '.content[] | .id + "," + .name'

      Example: results:


<Integration-ID>,embedded-ABX-onprem
<Integration-ID>,raas
<Integration-ID>,gss support
<Integration-ID>,embedded-VRO

Note: The <Integration-ID> can be retrieved from Assembler > Infrastructure > Connections > Integrations > embedded-VRO in the browser address bar as seen below, id taken after %2F



5. Review the Orchestrator Integration details:

       curl -k "https://$VRA/iaas/api/integrations/<Integration ID captured in step-4>?apiVersion=2021-07-15" -H "Authorization: Bearer $TOKEN" | jq .

6. Patch the integration using API:

  curl -v -k -X PATCH "https://$VRA/iaas/api/integrations/<Integration ID captured in step-4>?apiVersion=2021-07-15" -H "Accept: application/json" -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" --data-raw                         '{"integrationProperties": {"certificate":"Certificate content captured in step 3"}, "customProperties": {"certificate":"Certificate content captured in step 3"}}'

Body with just certificate:

{
   "integrationProperties": {
      "certificate": "Certificate content captured in step 3"
      },
   "customProperties ": {
      "certificate": "Certificate content captured in step 3"
      }
 }

 

7.After applying the certificate, check the Integration properties as in Step 5.

8. After the certificate has been applied, the data collection should recover, however in certain circumstances the same error may still be seen and require a recovery of either property: 

set empty for "vroUnresponsiveReason":""
or set  "vro-responsive": "true"

      1. Patch the integration using API:

        curl -v -k -X PATCH "https://$VRA/iaas/api/integrations/<Integration ID captured in step-3>?apiVersion=2021-07-15" -H "Accept: application/json" -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" --data-raw  '{"customProperties": {"vroUnresponsiveReason":""}}'

        Body with just vroUnresponsiveReason: 

{
  "customProperties":  
  {
    "vroUnresponsiveReason":""
  }
}

 

          9.If the update integration fails due to properties to be read-only, then please contact Broadcom support

Scripted Method

  1. Download and Transfer the Script Download the attached fix_vro.sh file from this article and transfer it to your VMware Aria Automation node using a file transfer tool like SCP or WinSCP.

  2. Make the Script Executable Log in to the Aria Automation node via SSH as the root user, navigate to the directory where you saved the script, and run the following command:

    Bash
     
    chmod +x fix_vro.sh

  3. Execute and Authenticate Run the script by executing ./fix_vro.sh. When prompted, provide a Bearer token for the impacted tenant (use the default tenant if multi-tenancy is not enabled in your environment).

    Note: If you are unsure how to get this, you can extract a Bearer token from your active browser session's developer tools, or generate one by following the instructions in KB 346005.

  4. Select the Target Integration The script will query the system and list all vRO registrations associated with the specified tenant. Locate the impacted vRO integration from the output and paste its ID into the prompt.

  5. Specify the vRO Hostname The script needs to know where to pull the new certificate from:

    • For External vRO: Type the exact hostname/FQDN of your external vRO appliance.

    • For Embedded vRO: The Aria Automation Load Balancer FQDN is provided as the default value. Simply press Enter to accept it.


6. Validate and Apply the Patch The script will fetch the Base64 certificate string from the designated vRO endpoint and display it on your screen. Review the printed certificate carefully. Once you validate that it is the correct certificate, type y at the prompt. The script will then proceed to patch the vRO endpoint via API to restore the connection.


         

7. On successful patching, you should see http 202 response as seen in the above screenshot. 

 

Attachments

fix_vro.sh get_app
Powered by Wolken Software