The internal certificate in vRealize Operations is generated upon initial deployment. Currently, upgrading to later versions of vRealize Operations does not upgrade the internal certificate.
Note: It is not possible or supported to replace the internal certificate with a custom certificate.
Resolution
Identify if Certificate Renewal is Required
First, validate is the certificate renewal is required. If the certificate is not yet expired, the certificate can be checked from a Web Browser. See the steps below for the most common web browsers:
If step 2 returns nothing, certificate renewal is not yet required.
If step 2 returned output containing validity check failed, certificate renewal is required immediately.
Certificate Renewal
To renew the certificate, install the applicable pak file to generate a new internal certificate. Depending on if the certificate has expired or not, choose the following applicable steps to install the PAK file.
If the vRealize Operations internal certificate has not yet expired, install the vRealize Operations Certificate Renewal PAK file while the vRealize Operations cluster is in an Offline state.
Note: Ensure all of the following steps are completed on all nodes in the vRealize Operations cluster unless noted otherwise.
If the admin UI after a long time does not report that installation of the Certificate Renewal PAK file as completed, complete the following steps.
Log into the Primary node as root via SSH or Console.
Run the following command to update the PAK installation status:
sed -i -e 's/\"initialization_state\"\:\"INITIALIZING\"/\"initialization_state\"\:\"NONE\"/g' /data/db/casa/webapp/hsqldb/casa.db.script
Repeat steps 1-2 on the Primary Replica node (if present).
Run the following command on the Primary and Primary Replica (if present) nodes to restart the CaSA service:
service vmware-casa restart
Internal Certificate Expired
If the vRealize Operations internal certificate has already expired, the vRealize Operations Certificate Renewal PAK file will need to be installed manually. Complete the following steps on the vRealize Operations cluster while the cluster is in an Offline state. Note: Ensure all of the following steps are completed on all nodes in the vRealize Operations cluster unless noted otherwise.
Note: Replace build with the build number of the downloaded vRealize Operations Certificate Renewal PAK file. Example: unzip /tmp/vRealize_Operations_Manager_Enterprise_Certificate_Renewal-8.0.0.15217416.pak -d /data/db/pakRepoLocal/vRealize_Operations_Manager_Enterprise_Certificate_Renewal/extracted
Stop all services by running the following commands:
service vmware-vcops-watchdog stop service vmware-vcops stop
Note: For versions 8.3 and later, ensure that all services have been stopped by running the vrops-status command. If there is a running service please kill it manually.
Example: (vpostgres) is running (3557) Run this command to terminate the process: kill -9 3557
Check if the is_admin property is set only for the Primary node in casa.db.srcipt.
On all nodes (including Remote Collectors and Witness) run the following command to verify the status of the is_admin property:
sed -nre "/clusterMembership/ s/^[^']+'([^']+)','([^']+)'.*/\2/p" /storage/db/casa/webapp/hsqldb/casa.db.script | python -m json.tool
In the output ""is_admin_node": true" should only be set when the "slice_name": "MASTER". If true is set for other nodes, complete the following on all nodes (including Remote Collectors and Witness):
Run service vmware-casa stop
Edit /storage/db/casa/webapp/hsqldb/casa.db.script and ensure "is_admin_node" is set to true for the Primary node, and false for all other nodes.
Run service vmware-casa start
The following command needs to be run in a particular order. Follow each sub-step carefully.
First, run the command on all Remote Collector nodes (if present) in the cluster, and wait for the task to complete. Continue to step 8.2.
Next, run the command on all Data nodes, the Witness node (if present), and the Primary Replica node (if present) in the cluster; do not wait for each node to complete, just start the command on all nodes. Once Waiting for certificate generation to complete appears on the last node, wait roughly 60 seconds, and continue to step 8.3.
Finally, run the command on the Primary node.
The expected behavior is for the command to finish, then shortly afterwards the pending tasks on the Data nodes and Primary Replica node (if present) will complete.
Note: To ensure that the command completes successfully check for the existence of the /var/vmware/_cert_generation_completed file on the Primary node.
Change newly generated certificates permissions on all nodes in the vRealize Operations cluster by running the following commands:
Log into the vRealize Operations Admin UI as the local admin user.
Click Bring Offline under Cluster Status.
If the cluster fails to go offline, click Force Offline under Cluster Status.
Note: Wait for the Cluster Status to show as Online.
Click Bring Online under Cluster Status.
Note: Wait for the Cluster Status to show as Online.
Additional Information
Note: After the certificate renewal, vRealize Operations retains the previous key in it's truststore once the new certificate is generated. vRealize Operations will use both the old and new certificate for validation. Currently there is no revocation mechanism.
Impact/Risks: The following instructions are for vRealize Operations 6.3 - 8.10.x. There is no recovery option for vRealize Operations Manager 6.2.x and earlier.
Upgrading to vRealize Operations 8.6 also upgrades the internal certificate during the upgrade process, except for setups with Cloud Proxies connected. The VMware vRealize Operations Certificate Renewal PAK file must be applied on vRealize Operations 8.6 only if the internal certificate has expired and if a Cloud Proxy node was connected to vRealize Operations before upgrade to 8.6.
If you are on vRealize Operations 8.5 or lower with Cloud Proxies connected, you must upgrade to vRealize Operations 8.6 before upgrading to vRealize Operations 8.10 or later. If you are on vRealize Operations 8.5 or lower, and you do not have Cloud Proxies connected, you can upgrade directly to vRealize Operations 8.10 or later, within the bounds of your Upgrade Path.
After upgrading the vRealize Operations certificate by PAK file, any Cloud Proxies certificates need to be upgraded manually (vRealize Operations 8.4 or later). This must be done by extracting the root certificate from vRealize Operations by using any browser and upload to Cloud Proxy by following Add CA certs while deploying a cloud proxy in vRealize Operations 8.4 or later.
Before following the resolution below, it is vital to snapshot the vRealize Operations nodes by following How to take a Snapshot of vRealize Operations. Note: If the certificate has already expired, and the Admin UI is not accessible, steps 1, 2, 10, and 11 can be skipped on the above mentioned KB.
Note: Upgrading vRealize Operations is not a replacement for this article. The steps below still must be followed.