vRealize Automation 7.x certificate process overview, common issues, and troubleshooting guidance

Products

VMware Aria Suite

Issue/Introduction

This article contains step by step instructions to update and/or replace all certificates in vRealize Automation 7.x with additional troubleshooting guidance and other known certificate product issues that may need to be corrected before continuing with product certificate updates.

See the "Updating vRealize Automation Certificates" section in the vRA 7.6 Documentation for additional information.

Note: End of General Support for VMware vRealize Automation 7.x

Environment

VMware vRealize Automation 7.x

Resolution

Prerequisites

Accounts and Credentials

root account for each vRA appliance
[email protected]
Domain / Local administrator access to each Windows domain joined IaaS component node

Files or Utilities

openssl
External or Internal CA to generate certificates
Access to any external connector appliances if using Smart Card authentication

Common expired certificate troubleshooting commands

To be used when manually validating the expiry of
- /storage/db/root.crt
- /storage/db/pgdata/server.crt
- /etc/apache2/server.pem
- /opt/vmware/etc/lighttpd/server.pem

openssl x509 -in /storage/db/pgdata/server.crt -text -noout

Known Issues and Additional Notes

If you are presented with HSTS warnings within your client browser when accessing any vRO or vRA web interface, it is recommended to secure the virtual appliances with publicly trusted certificates. Several public CAs offer free automatically renewable certificates.
- To bypass this warning type thisisunsafe anywhere in the warning page of Edge or Chrome. This should allow the web interface to load.
  - This behavior is similarly described under the following KB Error 'NET::ERR_CERT_INVALID' accessing HTML client for vRO 7.6
Error: connect: Connection refused seen when logging into the vRA Appliance management interface vRA > Certificates.
- To be used when manually updating a vRealize Automation 7.x IaaS Manager Service VMPS website certificate when not co-located with an IaaS Web component
```
netsh http show sslcert
netsh http update sslcert ipport=0.0.0.0:443 certhash=thumbprint appid='{########-####-####-####-############}'
```

Note: Update the certhash and appid values to match your environment.

*.cer and *.crt are pem formatted files with a different extension, one that is recognized by Windows Explorer as a certificate.

Note: *.crt and *.pem files extensions can be swapped to open the certificates in Windows

Smart Cards with external vIDM Connectors

If using Smart Card authentication, the following symptoms may be seen after replacing vRA certificates
- vRA application logs for the Event broker component, labeled as cafe:event-broker, located at /var/log/vmware/vcac/catalina.out contains messages similar to

com.vmware.vcac.eventlog.auditing.saveEvent:90 - Request to vCO failed.  Error: 400

vRO logins fail with messages similar to

The provided credentials are not valid.

The workflow in vRO Update a vRA host / Update the Host fails with a message similar to
```
Can not update the vRA host.
```

Updating & replacing all vRA 7.x certificates

Validate the internal vPostgres certificate is not expired: vRA 7.x APIs return error 500 when connecting to Postgres DB
Note: The above issue will prevent VAMI operations from completing as expected. Ensure this is not impacting your environment before moving forward as the proceeding commands will fail otherwise.
Note: Do not copy or store manually backed up configuration files within /storage/db or /storage/db/pgdata.
If ManagementAgent is about to expire when logged into the vRA VAMI vRA > Cluster tab: See the "Replace a Management Agent Certificate" section in the vRA 7.6 Documentation
Validate solution users in IaaS are not expired: At least one security token in the message could not be validated reported in Proxy Agent logfiles
See the "Replace Certificates in the vRealize Automation Appliance" section in the vRA 7.6 Documentation
"Replace the Infrastructure as a Service Certificate" section in the vRA 7.6 Documentation
"Replace the IaaS Manager Service Certificate" section in the vRA 7.6 Documentation
See the "Virtual Machine Template" section under "Updating vRealize Automation Certificates" in the vRA 7.6 Documentation for additional information on Guest Agent and Software Agent certificate update issues if templates are not updated.
See the "Update Embedded vRealize Orchestrator to Trust vRealize Automation Certificates" section in the vRA 7.6 Documentation

Note: This information is called out here:
The trust command syntaxes shown herein are representative rather than definitive. While they are appropriate for most typical deployments, there may be situations in which you need to experiment with variations on the commands.
- If you specify --certificate you must provide the path to a valid certificate file in PEM format.
- If you specify --uri, you must provide the uri from which the command can fetch a trusted certificate.
- If you specify the --registry-certificate option, you indicate that the requested certificate should be treated as the certificate for the component registry and the
- trusted certificate is added to the truststore under a specific alias used by the component registry certificate.
See the "Apply the new vRA / vRO certificate to the VAMI and Control Center web services" section in the vRA 7.6 Documentation for additional information.
Note: This certificate is shared, see Accessing vRealize Automation 7.x's Virtual Appliance Management Interface (VAMI) Best Practices.

cp /etc/apache2/server.pem /opt/vmware/etc/lighttpd/server.pem && service vami-lighttp restart && service haproxy restart
Note: The above will implement Step #8 and the documentation steps as one command.
Test accessing each appliance web interfaces over
- VAMI over port 5480
- vRA tenant portals over 443
- vRO Control Center over port 8283
- HTML5 vRO Client over port 8283
All web services should now share the same certificate thumbprint or be fully trusted with updated certificates.
Move onto the next section if using external vIDM connectors for Smart Card authentication.

Smart Card Configurations

Copy the vIDM connector certificate in pem format to the /tmp directory on the vRA appliance(s).

Trust the external connector appliance certificates by running the below command

/usr/local/horizon/scripts/installExternalSslRootCA.hzn --ca /path/to/certchain.pem --alias connector-root

Additional Information

If embedded vRO issues persist on a vRA appliance see: Resetting the Embedded vRealize Orchestrator 7.x configuration on a vRealize Automation 7.x appliance

For additional guidance in generating a signing request and Signing vRA certificates using an internal Microsoft CA signing authority.

Video guidance on generating a valid certificate signing request from custom configuration files with subject alternative name entries: vRealize 7.x – Replacing vRA Certificates

HSTS is HTTP Strict Transport Security: a way for sites to elect to always use HTTPS. See https://www.chromium.org/hsts.

NOTE: The vRA 7.6 documentation is no longer available online and must be downloaded and searched. The download can be found here: vRA 7.6 Documentation