Upgrade of Aria Operations hangs due an issue with Cached Roles

Products

• VCF Operations/Automation (formerly VMware Aria Suite)
• VCF Operations

Issue/Introduction

There are situations where the CACHED ROLES document can have the primary node's Admin role removed in HA/CA clusters /storage/db/casa/webapp/hsqldb/casa.db.script file.

Manually editing this file is risky and prone to mistakes causing additional cluster issues. The script mentioned in this article is used to detect and correct the cached role values automatically.

This issue has been faced during cluster maintenance processes (online, offline, etc.) and during the upgrade process.

This issue could cause different problems in an Aria Operations cluster. This problem can be faced during the upgrade process, and because of the invalid roles in the cached_roles document, analytics is not able to run successfully. This causes the entire upgrade process to fail, or you may find that the upgrade hangs.

Error in the /storage/log/vcops/log/casa/casa.log similar to below:

INFO [ajp-nio-xxx.x.0x.x-8011-exec-2] [xxxxxxxx] support.subprocess.GeneralCommand:255 - Command '/usr/bin/sudo -n /usr/lib/vmware-python-3/bin/python /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py --action new_validate --pak vRealizeOperationsManagerEnterprise-818324521385 --json --force_content_update false --roles ADMIN,DATA,UI' threw exception: CommandLineExitException: key=general.failure; args=1,; cause=
WARN [ajp-nio-xxx.x.0x.x-8011-exec-2] [xxxxxxxx] casa.exception.CasaControllerExceptionHandler:212 - cause for exception = CommandLineExitException: key=general.failure

Another example is an "Inventory sync" failure in VMware Aria Suite Lifecycle with Error code LCMVROPCONFIG20066, which uses the casa API to get nodes roles. You will not see the Aria Operations node having invalid roles in the cached_roles document in VMware Aria Suite Lifecycle under Aria Operations Environment.

Another example is an "Inventory sync" failure in VMware Aria Suite Lifecycle with Error code LCMVROPSYSTEM25026.

If the /casa/cluster/status API returns a non-valid role for one of the nodes (most likely for the master node), this means that you have faced this issue.

Environment

• VMware Aria Operations 8.x
• VMware Cloud Foundation Operations 9.x

Cause

The root cause of this failure is currently unknown.

Resolution

To understand that we have this problem, we need to check the CACHED_ROLES document on each cluster member.

1. Take the cluster offline and take snapshots of all nodes in the cluster as per KB How to take a Snapshot of VMware Aria Operations.
2. Reboot the cluster nodes as per KB Shutdown and Startup sequence for Aria Operations cluster.
3. Download the getCachedRoles.py and restoreCachedRoles.py scripts attached to this article.
4. Copy the script files to the /tmp directory using an SCP utility like WinSCP.
5. Log in to an SSH session as root to the primary node.

Note: The vmware-casa service must be running for the scripts to function properly. Run the command service vmware-casa status to validate.

6. Run the command $VMWARE_PYTHON_3_BIN getCachedRoles.py to dump the CACHED_ROLES document from all nodes. The result can be found in the same directory with the name cachedRoles.json. Example of a valid cluster:

   > Primary: ADMIN, DATA, UI
   > Primary Replica : ADMIN, DATA, UI, REPLICA
   > Data. : DATA, UI
   > Remote Collector : REMOTE_COLLECTOR
   > Witness : WITNESS

7. Review the file to determine if the ADMIN role is removed from the Primary or Primary Replica node.

Note: The first entry in the file is typically the Primary node and lists what cachedRoles information it has for itself and the remaining nodes in the cluster. The next section would have the next node in the cluster such as the Replica and what it has for cachedRoles for the Primary node, itself, and then all other nodes.

8. If the ADMIN role is missing on the Primary or Primary Replica node, run the command $VMWARE_PYTHON_3_BIN restoreCachedRoles.py --restore to restore the CACHED_ROLES document, which should fix the roles.
9. Try to then start the analytics service if it was not running, or resume any cluster online, upgrade, or sync operations to determine if the issue is now resolved.