Upgrade to vRealize Operations 8.6.x fails at firstboot phase

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Upgrading to vRealize Operations 8.6.x fails at the firstboot phase.
• The casa service won't start.
• The logs/pakManagervcopsPakManager.root.post_apply_system_update.log file shows errors similar to:

...
2022-03-22T05:44:23 ERROR [8736] - root - [Failed] /usr/lib/vmware-casa/firstboot/casa-firstboot.py casa-firstboot.py - Failed admin-app-firstboot.py
...
2022-03-22T05:44:23,711+0000 [9871] - root - Failed to find the key: "default.store_type" in the file: "/storage/vcops/user/conf/ssl/secure-communications.properties" Failed to load "default.store_type" property from the configuration file: /storage/vcops/user/conf/ssl/secure-communications.properties

Environment

VMware vRealize Operations 8.0.x
VMware vRealize Operations 8.2.x
VMware vRealize Operations 8.6.x
VMware vRealize Operations 8.5.x
VMware vRealize Operations 8.3.x
VMware vRealize Operations 8.1.x
VMware vRealize Operations 8.4.x

Resolution

To resolve this issue, replace the communications.properties.casa-b2b.bkp created by the upgrade process during the upgrade.
This can be complete manually, or through the use of a script.  Complete the steps outlined in the method you choose.

Quick Links:

• Manual Method
• Script Method

Manual Method

Complete the following on all Analytics nodes (Primary, Replica (if present), and Data nodes) simultaneously.

1. Ensure snapshots have been created per the Impact/Risks section of this article.
2. Log into the Analytics node as root via SSH or Console.
3. Run the following command to backup the secure-communications.properties file:

cp /storage/vcops/user/conf/ssl/secure-communications.properties /storage

4. Start the upgrade process and monitor the /storage/vcops/user/conf/ssl/ directory for creation of the communications.properties.casa-b2b.bkp file.
5. Once the communications.properties.casa-b2b.bkp has been created, run the following command to replace the file with the backup created in step 3:

cp /storage/secure-communications.properties /storage/vcops/user/conf/ssl/communications.properties.casa-b2b.bkp

The upgrade should continue and complete normally.
Note: If you are not quick and the system has rebooted before replacing the communications.properties.casa-b2b.bkp file, then revert back to the snapshots created in step 1 and try again.

 

Script Method

Complete the following on all Analytics nodes (Primary, Replica (if present), and Data nodes) simultaneously.

1. Ensure snapshots have been created per the Impact/Risks section of this article.
2. Download the b2bfilefix.sh file from the Attachments section on the right of this article.
3. Using an SCP utility, upload the b2bfilefix.sh file to the /tmp directory on all Analytics nodes in the cluster.
4. Log into the node as root via SSH or Console.
5. Run the following command to set the owner of the b2bfilefix.sh file to root, and make it executable:

chown root /tmp/b2bfilefix.sh && chmod +x /tmp/b2bfilefix.sh

6. Run the following command to start the b2bfilefix.sh script:

./tmp/b2bfilefix.sh

7. Start the upgrade process and wait for completion.

Additional Information

Impact/Risks:
Take snapshots following How to take a Snapshot of VMware Aria Operations before proceeding with the Resolution section.