Upgrade to vRealize Operations 8.10.x hangs due to unexpected allowAllResources attribute in auth table

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Upgrade to vRealize Operations 8.10.x appears to hang due to Analytics service restarts.  The restarts are caused by the existence of an invalid allowAllResources attribute in some UserWrapper rows in the postgres kv_auth table.
• The /storage/log/vcops/log/analytics-wrapper.log show errors similar to:

2023/03/06 06:15:28 | INFO   | jvm 1    | >>> AnalyticsMain.run failed with error: AuthException: Auth Exception occurred : Auth Exception occurred : Auth Exception occurred : No such field com.vmware.vcops.auth.xml.bean.UserWrapper.allowAllResources
2023/03/06 06:15:28 | INFO   | jvm 1    | ---- Debugging information ----
2023/03/06 06:15:28 | INFO   | jvm 1    | message             : No such field com.vmware.vcops.auth.xml.bean.UserWrapper.allowAllResources
2023/03/06 06:15:28 | INFO   | jvm 1    | field               : allowAllResources
2023/03/06 06:15:28 | INFO   | jvm 1    | class               : com.vmware.vcops.auth.xml.bean.UserWrapper
2023/03/06 06:15:28 | INFO   | jvm 1    | required-type       : com.vmware.vcops.auth.xml.bean.UserWrapper
2023/03/06 06:15:28 | INFO   | jvm 1    | converter-type      : com.thoughtworks.xstream.converters.reflection.ReflectionConverter
2023/03/06 06:15:28 | INFO   | jvm 1    | line number         : -1
2023/03/06 06:15:28 | INFO   | jvm 1    | version             : 1.4.20
2023/03/06 06:15:28 | INFO   | jvm 1    | -------------------------------
2023/03/06 06:15:28 | INFO   | jvm 1    | WrapperManager Debug: WrapperManager.stop(-1) called by thread: SystemExitThread

• The /storage/log/vcops/log/analytics-GUID.log show errors similar to:

2023-03-06T06:15:28,299+0000 ERROR [Analytics Main Thread]  com.vmware.vcops.auth.server.AuthorizationServer.setUserAuthServerDetails > - Auth Exception occurred : No such field com.vmware.vcops.auth.xml.bean.UserWrapper.allowAllResources
---- Debugging information ----
message             : No such field com.vmware.vcops.auth.xml.bean.UserWrapper.allowAllResources
field               : allowAllResources
class               : com.vmware.vcops.auth.xml.bean.UserWrapper
required-type       : com.vmware.vcops.auth.xml.bean.UserWrapper
converter-type      : com.thoughtworks.xstream.converters.reflection.ReflectionConverter
line number         : -1
version             : 1.4.20
-------------------------------
com.vmware.vcops.auth.exception.AuthException: Auth Exception occurred : No such field com.vmware.vcops.auth.xml.bean.UserWrapper.allowAllResources

Environment

VMware vRealize Operations 8.6.x
VMware vRealize Operations 8.3.x
VMware vRealize Operations 8.10.x
VMware vRealize Operations 8.4.x
VMware vRealize Operations 8.5.x
VMware vRealize Operations 8.2.x

Cause

The allowAllResources JSON attribute has been removed from the UserWrapper class in vRealize Operations 8.10.x.
In some scenarios the Cassandra auth data migration process may not remove this attribute during the upgrade.

Resolution

Follow one of the sets of steps below depending on the current cluster status.

• Upgrade in progress - Follow these steps if the issue was discovered during the upgrade process.
• Upgrade not started - Follow these steps if you have not yet started the upgrade, or you have restored the cluster to a pre-upgrade status.

Upgrade in progress

1. Log into the Primary node as root via SSH or Console.
2. Run the following command to remove the allowAllResources attribute from all entries in the kv_auth table:

su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -d vcopsdb -p 5433 -c \"UPDATE kv_auth SET col__kv_strvalue = (col__kv_strvalue::jsonb) #- '{UserWrapper,allowAllResources}' WHERE objecttype ='User-'  AND (col__kv_strvalue::jsonb)->'UserWrapper' ? 'allowAllResources';\""

The upgrade will continue on it's own.  Return to the Admin UI and wait for the upgrade to complete. 

 

Upgrade not started

1. Download the attached auth_UserWrapper_allowAllResources_cassandra.py script from the Attachments section on the right of this knowledge article.
2. Using an SCP utility, copy the auth_UserWrapper_allowAllResources_cassandra.py script to the /tmp directory on the vRealize Operations Primary node.
3. Log into the Primary node as root via SSH or Console.
4. Run the following command to move to the /tmp directory:

cd /tmp

5. Run the following command to grant to read permission to the auth_UserWrapper_allowAllResources_cassandra.py script:

chmod 744 auth_UserWrapper_allowAllResources_cassandra.py

6. Run the following command to dump the existing Cassandra auth table to a CSV file:

$VMWARE_PYTHON_BIN /usr/lib/vmware-vcops/cassandra/current/bin/cqlsh.py --ssl --cqlshrc /usr/lib/vmware-vcops/user/conf/cassandra/cqlshrc -e "COPY globalpersistence.auth TO '/tmp/cassandra_auth.csv' WITH NULL='<##NULL##>' AND HEADER=TRUE;"

7. Execute the auth_UserWrapper_allowAllResources_cassandra.py script to create before/after delta files with/without the allowAllResources attribute:

$VMWARE_PYTHON_BIN /tmp/auth_UserWrapper_allowAllResources_cassandra.py

8. Compare the /tmp/cassandra_auth.AffectedRowsBefore.csv and /tmp/cassandra_auth.AffectedRowsAfter.csv delta files to ensure only the UserWrapper|allowAllResources entries were removed.
9. Run the following command to update the auth table using the /tmp/cassandra_auth.AffectedRowsAfter.csv file:

$VMWARE_PYTHON_BIN /usr/lib/vmware-vcops/cassandra/current/bin/cqlsh.py --ssl --cqlshrc /usr/lib/vmware-vcops/user/conf/cassandra/cqlshrc  -e "COPY globalpersistence.auth FROM '/tmp/cassandra_auth.AffectedRowsAfter.csv' WITH NULL='<##NULL##>' AND HEADER=TRUE;"

You can now start the vRealize Operations 8.10.x upgrade as normal.

Additional Information

Impact/Risks:
It is recommended to take a snapshot of all nodes before proceeding with the resolution step by following How to take a Snapshot of VMware Aria Operations.

Note: If you are unable to take the cluster offline, follow Shutdown and Startup sequence for a VMware Aria Operations cluster.