VCF Operations upgrade from 9.0.1.0 to 9.1.0.0 appears to complete but then restarts and stops without completing

Products

• VCF Operations

Issue/Introduction

• The upgrade completes 14 of 14 steps initially
• The upgrade restarts again (without any manual intervention to cause the restart) and stops at step 3 of 14
• The VCF Operations nodes (primary, primary replica, data) appear to have been upgraded to 9.1.0.0
• The Collector node (Cloud Proxy) is still running 9.0.1.0, it has not been upgraded to 9.1.0.0

Environment

VCF Operations 9.1

Cause

Unknown, however the VCF Operations cluster may potentially be in an inconsistent state - which is blocking the upgrade.

Resolution

1. Take snapshots of all VCF Operations nodes - but leave the cluster offline - do not bring it online just yet (as per Snapshot Creation in VMware Aria Operations)
2. With the cluster still offline, login to the Primary node over SSH and run the following command block:

   echo -e "\e[1;32mCasa.db.script edit initiated\e[0m";service vmware-casa stop;sleep 10;cp --backup=t /storage/db/casa/webapp/hsqldb/casa.db.script /storage/db/casa/webapp/hsqldb/casa.db.script.backup;sed -ri 's/"onlineState":"\w+"/"onlineState":"OFFLINE"/g;s/"initialization_state":"\w+"/"initialization_state":"NONE"/g;s/"online_state":"\w+"/"online_state":"OFFLINE"/g;s/"online_state_reason":"\w+"/"online_state_reason":""/g;s/"remove_node_state":"\w+"/"remove_node_state":"NONE"/g;s/"installation_state":"\w+"/"installation_state":"DONE"/g' /storage/db/casa/webapp/hsqldb/casa.db.script;sleep 2;service vmware-casa start;echo -e "\e[1;32mCluster Status has been modified\e[0m";grep "onlineState" /storage/db/casa/webapp/hsqldb/casa.db.script;

3. Repeat step 2 on each other Analytics node in the cluster (Replica and Data nodes)
4. From the Admin UI, bring the cluster back online again.
5. Reapply the VCF Operations 9.1 upgrade to VCF Operations again
6. The issue is then expected to be resolved, VCF Operations nodes (primary, primary replica, data) and the cloud proxies are expected to be running 9.1