Error Code: LCMVROPCONFIG20035 VMware Aria Operations water mark configurations failed as master node hostname is empty. Check whether VMware Aria Operations cluster is up and running.

Products

• VMware Aria Suite

Issue/Introduction

After upgrading Aria Operations using Aria Suite Lifecycle, the upgrade failed to complete with an error:

Error Code: LCMVROPCONFIG20035
VMware Aria Operations water mark configurations failed as master node hostname is empty. Check whether VMware Aria Operations cluster is up and running.
Failed while creating SDDC Manager water mark for VMware Aria Operations as master node itself is not present, Check whether VMware Aria Operations cluster is in proper state.

However, when checking Aria Operations admin UI, cluster is online and upgraded successfully.

The only error now is that the incorrect version is being reported in Life Cycle Suite and no version update report was pushed down to SDDC manager, Inventory sync also fails with same Error.

Environment

Aria Operations Version : 8.18.3
Life Cycle Suite : 8.18.0.0 Patch Pack 5
VCF: 5.2.1.0

Cause

During cluster maintenance processes or upgrade processes, Cluster "CACHED ROLES" document is broken and Master/Admin role is removed from the Master node in a HA/CA cluster.

Resolution

To understand the problem, we need to check CACHED_ROLES document on each node of the cluster.

To get Cached Roles from all nodes:

• Copy getCachedRoles.py from the attachments of this KB to '/tmp' directory on one of the nodes of the cluster using a scp tool like winscp.
• Use the below commands to run the file.
  

  cd /tmp
  python3 getCachedRoles.py

• Check cachedRoles.json generated file (in the /tmp directory), you will notice "role": "MASTER" missing

To re-generate Cached roles doc and distribute to all members: 

• Copy  restoreCachedRoles.py from the attachments of this KB to '/tmp' directory on one of the nodes of the cluster.
  
  
• Use the below commands to run the file.
  

  cd /tmp
  python3 restoreCachedRoles.py

Once the restoreCachedRoles.py is completed and cached roles are fixed in getCachedRoles.py. Inventory sync in  Aria Suite Lifecycle should work.