vRealize Operations upgrade from 8.6.4 to 8.10 fails due to a missing Cassandra table "vmwarecloudonawscomputationratio"

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Errors similar to the following can be seen in the /storage/log/vcops/log/cassandradbupgrade.log file:

2022-12-01T21:42:00,954+0000 ERROR [main]  .runOneScript - Exception running upgrade script:834
java.lang.reflect.InvocationTargetException: null
 at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
 at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
 at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
 at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
 at com.vmware.vcops.dbupgrade.cassandra.CassandraDbUpgrade.runOneScript(CassandraDbUpgrade.java:250) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.dbupgrade.cassandra.CassandraDbUpgrade.upgradeDB(CassandraDbUpgrade.java:414) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.dbupgrade.cassandra.CassandraDbUpgrade.upgradeCassandraDB(CassandraDbUpgrade.java:116) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.dbupgrade.cassandra.CassandraDbUpgrade.main(CassandraDbUpgrade.java:562) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
Caused by: com.vmware.vcops.kvstore.error.DBException: Unable to getSafe for future: com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured table vmwarecloudonawscomputationratio
 at com.vmware.vcops.kvstore.cassandra.CassandraClient.getSafe(CassandraClient.java:473) ~[vcops-kvstore-cassandra-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.kvstore.cassandra.CassandraClient.safeExecute(CassandraClient.java:381) ~[vcops-kvstore-cassandra-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.kvstore.cassandra.CassandraClient.executeRead(CassandraClient.java:339) ~[vcops-kvstore-cassandra-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.kvstore.cassandra.CassandraClient.executeRead(CassandraClient.java:331) ~[vcops-kvstore-cassandra-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.dbupgrade.cassandra.migration.CostDataMigration.execute(CostDataMigration.java:76) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
 ... 8 more

• In the vRealize Operations UI, if you attempt to browse to Configure > Cost Settings > Settings, this pop up will fail to load if this table is missing.

Environment

VMware vRealize Operations 8.6.x
VMware vRealize Operations 8.10.x

Resolution

1. Revert all nodes to snapshots taken before the upgrade attempt.
2. Log into the Primary node as root via SSH or Console.
3. Connect to Cassandra using the following command:

$VMWARE_PYTHON_BIN /usr/lib/vmware-vcops/cassandra/apache-cassandra-3.11.12/bin/cqlsh --ssl --cqlshrc /usr/lib/vmware-vcops/user/conf/cassandra/cqlshrc

4. Switch to the globalpersistence schema, re-create the table and insert default data.  Run the following commands in the given order:

• USE globalpersistence;
• CREATE TABLE if not exists globalpersistence.vmwarecloudonawscomputationratio ( vmwarecloudonawscomputationratio_id varchar, memory double, cpu double, storage double,  currency varchar, tenant_id varchar, last_update bigint, qualifier map<varchar,varchar>, primary key (tenant_id,last_update,vmwarecloudonawscomputationratio_id) )with clustering order by (last_update desc) AND bloom_filter_fp_chance = 0.01 AND caching = {'keys':'ALL', 'rows_per_partition':'NONE'} AND comment = '' AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32'} AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE';
• INSERT INTO vmwarecloudonawscomputationratio(tenant_id,last_update,cpu,memory,storage,currency,vmwarecloudonawscomputationratio_id) values('__default__',1672831768525,5,1,0.5,'USD','3af5f15f-37b1-4e3c-aafb-baf4665cfcb9');

5. Refresh the vRealize Operation UI, the popup for Configure > Cost Settings > Settings should now load correctly and you can proceed with the next upgrade attempt.