vSphere Cloud Account certificate is changed causing errors such as Unavailable for Deployment and Failed to validate

Products

VMware Aria Suite

Issue/Introduction

Symptoms:

You have recently changed the vSphere certificate that is used as a Cloud Account in VMware Aria Automation SaaS / 8.x (formerly VMware vRealize Automation 8.x)
You notice errors similar to "Unavailable for Deployment or Failed to validate" when attempting to update the Cloud Account.
You notice the following one of the following errors or similar when validating and saving the Cloud Account:
"Unable to update endpoint of type vsphere with hostname: vcenter-server.example.com ClientResponse has erroneous status code: 400 Bad Request"
You notice errors similar to "Cannot find available cloud zones in the current project. Maybe there is no connectivity to the endpoint(s)" when trying to provision.
You see error like the following in provisioning-service-app.log:
"ERROR provisioning [host='provisioning-service-xxx' thread='xxx' user='xxx' org='xxx' trace='xxx' parent='xxx' span=xxx'] c.v.p.c.m.a.v.VSphereEndpointAdapterService.lambda$validate$8:280 - handleCertificate serviceErrorResponse: {"message":"Error storing certificate. Retry validation","statusCode":400,"errorCode":0,"serverErrorId":"xxx","documentKind":"com:vmware:xenon:common:ServiceErrorResponse"} for host: xxx"
You have recently upgraded to Aria Automation 8.18.0 and now are unable to connect to the remote console of vCenter VMs through Automation.

Environment

VMware Aria Automation 8.x
VMware vRealize Automation 8.x

Cause

When a vSphere certificate is changed, the Aria Automation user revalidates the Cloud account and accepts the new certificate but closes the Cloud account without saving. The certificate is then accepted in the trust store but not saved to the endpoint.

Resolution

This issue is resolved in the future release of VMware Cloud Foundation 9.0.

See the Workaround section below for additional details.

Workaround:

Prerequisites

Please take simultaneous non-memory snapshots of each virtual appliance(s) in the cluster.
- Use the quiesce option for versions 8.9.x and above.
This article provides the option to update the Cloud Account using an Automation Orchestrator package and a manual curl command from the CLI instead of using Postman.
- If using the CLI option, validate you have SSH access with root username and password to at least one of the appliances in the cluster.

Procedure #1: Automating this process with Automation Orchestrator

Requirements:

Aria Automation Orchestrator plugin for Aria Automation

Steps:

Download the attached Automation Orchestrator package net.broadcom.vra.update.ca.certificate
Import this package into Orchestrator. Import a Package in the Automation Orchestrator Client
Create an Aria Automation plugin host with tenant administrator credentials. (This step can be skipped if you already have an Aria Automation plugin host for the impacted Automation Cluster) Add a VMware Aria Automation host
Start the workflow Update Cloud Account Certificate
Select the Aria Automation host from Step#3 for field Tenant Admin Host.
Once the host is selected, all configured vSphere cloud accounts will be populated in the vSphere Cloud Accounts to Update data grid.
Remove any cloud accounts which do not require a certificate update or leave the field Administrator password empty.
Enter data for the Administrator password field.
Trigger the workflow run and wait until the execution completes.
Once the workflow execution is completed you will see a summary of information for the update certificate operation in the workflow execution log.
Close the Cloud Account if it is open in the UI, then Validate the health of the Cloud Account. This may take up to 10 minutes to complete the data collection.

Procedure #2: Manual

Collecting vCenter Cloud Account Information

Obtain the Cloud Account ID and the username (vCenter Service Account) from the UI by opening the respective vCenter within the Aria Automation UI.

Alternatively, using Postman or cURL, search for the cloud account id by using: GET /iaas/api/cloud-accounts
Alternatively, you can get the cloud account ID from the browser by going to Developer tools (F12 in most browsers) > Network tab, refresh the Cloud Account page and then Ctrl-F to search for /iaas/api/cloud-accounts.

On Aria Automation SaaS: capture the Cloud proxy ID for use as dcid in the later payload.
Capture the vCenter certificate in PEM format by running the following command:

openssl s_client -connect ID1:443 2> /dev/null | openssl x509 | awk 'NF {sub(/\r/, "");  printf "%s\\n",$0;}'

Note: Replace ID1 with the vCenter FQDN.
Note: This exact output will be used later.

Obtain a Bearer token following the article Generate Access_Token/Bearer_Token in VRA 8.x for API Authentication or by using the browser DevTools using the next steps:
1. Access Aria Automation from a web browser and login
2. Open the web browser's Developer Tools > Network then copy the access bearer token.

After capturing this information, proceed to the Standalone or vCF-enabled sections Procedure sections below to update the Cloud Account in Aria Automation.

Procedure for Aria Automation SaaS and 8.x

Using Postman

Add the bearer in the authorization tab.
Add the following API call:

PATCH ID3/iaas/api/cloud-accounts/ID2?apiVersion=2021-07-15

In the body tab add the following information and then run the API call.

     {
         "privateKeyId": "ID4",
         "privateKey": "ID5",
         "certificateInfo": {
             "certificate": "ID6"
         }
     }

Note: Please replace the IDx values considering:

Replace ID2: with the cloud account id obtained in step 1 of the prerequisites.
Replace ID3: with the Aria Automation FQDN.
Replace ID4: with the username obtained in step 1 of the prerequisites.
Replace ID5: with the password of the vCenter service account or username (ID4)
Replace ID6: with the output of the command run in step 3 of the prerequisites.
Replace ID7: with the bearer token obtained in step 4 of the prerequisites under the authorization tab.

Note: For SaaS, the Cloud Proxy ID that maps to this cloud account will be required as dcid in the payload. This also applies to the Using cURL section below.

     {
         "privateKeyId": "ID4",
         "dcid": "CloudProxyID",
         "privateKey": "ID5",
         "certificateInfo": {
             "certificate": "ID6"
         }
     }

Close the Cloud Account if it is open in the UI, then validate the health of the Cloud Account. This may take 10 minutes to finish the data collection.

Using cURL

SSH to the appliance and run the following curl command:

curl -k --location --request PATCH 'https://ID3/iaas/api/cloud-accounts/ID2?apiVersion=2021-07-15' \
--header 'Authorization: Bearer ID7' \
--header 'Content-Type: application/json' \
--data-raw '{
    "privateKeyId": "ID4",
    "privateKey": "ID5",
    "certificateInfo": {
        "certificate": "ID6"
    }
}'

Note: Please replace the IDx values considering:

Replace ID2: with the cloud account id obtained in step 1 of the prerequisites.
Replace ID3: with the Aria Automation FQDN.
Replace ID4: with the username obtained in step 1 of the prerequisites.
Replace ID5: with the password of the vCenter service account or username (ID4). (Note: If using characters like quotes be sure to escape them in the curl command)
Replace ID6: with the output of the command run in step 3 of the prerequisites.
Replace ID7: with the bearer token obtained in step 4 of the prerequisites.

Close the Cloud Account if it is open in the UI, then validate the health of the Cloud Account. This may take 10 minutes to finish the data collection.

Procedure for vCF-enabled Aria Automation instances

GET the necessary cloud account details:

GET ID3/provisioning/uerp/provisioning/mgmt/endpoints?expand

Note: Use a valid bearer token for authorization

Find the relevant VCF endpoint and look for customProperties. Copy the relevant nsx and "vsphere" property starting with "resources/endpoints/...". Use these to make the following GET calls to check those cloud accounts in Step 2.

Check each of the NSX and vSphere cloud accounts using the links copied in Step 1
```
GET ID3/provisioning/uerp/resources/endpoints/xxx}
```

If the certificate doesn't match the valid certificate for that endpoint (you would have got that from running the command : openssl s_client -connect ID1:443 2> /dev/null | openssl x509 | awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' earlier), we will need to patch this cloud account.

Patch cloud account with an updated certificate

PATCH ID3/provisioning/uerp/resources/endpoints/xxx}
{
     "endpointProperties": {
          "certificate": "updated-endpoint-certpem"
     },
     "customProperties": {
          "certificate": "updated-endpoint-certpem"
     }
}

Finally update the vCF cloud account itself. certificate should be updated to reflect the updated vSphere certificate, and nsxCertificate should refer to the NSX endpoint certificate. Follow step 3 to make this change.

Note: Replace ID1 with the vCenter FQDN

Note: Replace ID3: with the Aria Automation FQDN.

Note: It is recommended to use Postman for ease of use, but the equivalent cURL commands are noted below. (Replace <bearer token> with the bearer token you obtained and replace <updated-endpoint-certpem> with the new certificate obtained from the endpoint you are updating.)

curl -k --location -g --request GET 'https://ID3/provisioning/uerp/provisioning/mgmt/endpoints?expand' \ --header 'Authorization: Bearer <bearer token>'

curl -k --location -g --request PATCH 'https://ID3/provisioning/uerp/resources/endpoints/xxx' --header 'Authorization: Bearer <bearer token>' --header 'Content-Type: application/json' --data-raw '{ "endpointProperties": { "certificate": "<updated-endpoint-certpem>" }, "customProperties": { "certificate": "<updated-endpoint-certpem>" }}'

Note: Replace the newline characters in the certpem with /n - the certpem should be entirely on one line.
Note: Patch the "NSX", "vSphere" and "VCF" endpoints if it's a VCF enabled Aria Automation Instance.

Note: Please Check "Additional Information" Section

Additional Information

Important note 1:

For procedures Using cURL

If you encounter an issue after running above steps

"message": "Failed to execute handlePatchAsync", "statusCode": 400,

Check the format of (ID4: with the username obtained in step 1 of the prerequisites) and match format UPN Domain@username.

Important note 2:

Situation:
In case Cloud account still do not update with the latest certificate although API call succeeds with 202 status code and "INPROGRESS".
Use the cloud-account-vsphere API along with the body as mentioned below:

/iaas/api/cloud-accounts-vsphere/

{
"hostName": "vCenterFQDN",
"certificateInfo": {
"certificate": " -----BEGIN CERTIFICATE-----\nMIIEeDCCxxxxxxxxxxxxxxxxxxxxxxxxxxAMIGkMRYwFAYD\nVQQDDA1xNHVzc3RzeXUwNTIwMRcwFQYKCZImiZPyLGQBGRYHdnNwaGVyZTEVMBMG\nCgmSJomT8WjQhsY\n0IvKGAGvZANwwMpVYyfEP/r5ugHE5XGzqnHtshP6cGO7Td9SHJpwzMezj8M=\n-----END CERTIFICATE-----\n "
},
"password":"password",
"username":"username@domain"
}

Attachments

net.broadcom.vra.update.ca.certificate.package get_app