Aria Operations Primary Node showing as Data Node in Aria Life Cycle Manager

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Aria Suite Lifecycle Manager is displaying Aria Operations nodes incorrectly leading to issues such as:

• While trying to perform inventory sync for Aria Operations, it failed with an error:

Injecting task failure event. Error Code : 'LCMVROPCONFIG20001', Retry : 'true', Causing Properties : '{ CAUSE :: vropsMasterNodeIP ===  }' 
com.vmware.vrealize.lcm.plugin.common.vrops.exceptions.VropsAddressException: Invalid VMware Aria Operations master address
at com.vmware.vrealize.lcm.plugin.core.vrops.tasks.BaseConfigureVropsTask.init(BaseConfigureVropsTask.java:71) [vmlcm-vropsplugin-core-8.16.0-SNAPSHOT.jar!/:?]

• Inventory sync works correctly but upgrading Aria Operations from Aria Suite Lifecycle Manager fails because it could not locate the primary IP for the Aria Operations master node.

 

Environment

• Aria Operations 8.18.x
• Aria Lifecycle Manager 8.18.x

Cause

If the roles are not correctly populated In the case.db.script, sync will fail with the error Invalid VMware Aria Operations master address.

The incorrect roles details is shown below. Instead of showing all the roles i.e., ADMIN","DATA","UI it will only show "DATA","UI"

INSERT INTO CASA_DOCS VALUES('CACHED_ROLES','{"document_time":epoch_time,"document_name":"CACHED_ROLES","document_version":1085156,"document_body":{"cachedRoles":[{"nodeIdentifier":"##########-####-####-####-###########","cachedRoles":["DATA","UI"]},{"nodeIdentifier":"##########-####-####-####-###########","cachedRoles":["ADMIN","DATA","UI","REPLICA"]},{"nodeIdentifier":"##########-####-####-####-###########","cachedRoles":["DATA","UI"]}]}}')

Resolution

1. Take the Aria Operations cluster OFFLINE from the Admin UI.
2. Take snapshots of all Analytics nodes in the cluster following instructions as described in Snapshot Creation in VMware Aria Operations
3. SSH to all the analytics nodes ( Cloud Proxy changes are not required)
4. Stop the casa service on all the nodes by running the below command

service vmware-casa stop

5. Edit casa.db.script (/data/db/casa/webapp/hsqldb) on all the nodes adding "ADMIN", where it is missing from the CACHED_ROLES section.
6. Increase the document_version count by 1 (e.g., the below script shows "document_version":1085157. Increase it to 1085158)

VALUES('CACHED_ROLES','{"document_time":epoch_time,"document_name":"CACHED_ROLES","document_version":1085157,"document_body":{"cachedRoles":[{"nodeIdentifier":"##########-####-####-####-###########","cachedRoles":["DATA","UI"]},{"nodeIdentifier":"##########-####-####-####-###########","cachedRoles":["ADMIN","DATA","UI","REPLICA"]},{"nodeIdentifier":"##########-####-####-####-###########","cachedRoles":["ADMIN","DATA","UI"]}]}}')

7. Start the casa service on all nodes by running the below command

service vmware-casa start

8. Bring the Aria Operations cluster ONLINE from the Admin UI.