Node status changing to offline /Going Online /waiting on analytics after cluster initialization

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Symptoms:

• The Aria Operations Cluster Status shows as Online/ Going Online after Aria Operations Upgrade 
• Node statuses will constantly swap between Online, Waiting on Analytics, and Offline while services remain online and functional
• If vRealize Operations HA is enabled, the HA status will show as Degraded
• The /storage/vcops/log/api.log file contains errors similar to the beneath:

[Timestamp],767 [ajp-nio-12x.xx.xx.x-8010-exec-1, amhIkx04tLi48M2HNeneSagVW7DmAewX] ERROR service.impl.ControllerFacadeFactory$ControllerFacadeInvocationHandler - Controller API call 'getShardRemoveProgress' has failed! Reason = [Federation call is not implemented.]

• The latest analytics log in /storage/vcops/log/ contains errors similar to the beneath:

[Timestamp],338+0000 ERROR [ServerConnection on port 10000 Thread 1] [ohYuLzDeu1a5OKEQtyUPIbBlTEZBUqp9] com.vmware.vcops.controller.BaseExecutor.executeWithPagination - getCollectors failed: Federation call is not implemented.
[Timestamp],594+0000 ERROR [ServerConnection on port 10000 Thread 1] [ekIGRfJ7xGD1cGZYrStzB7IxdjwJFWHs] com.vmware.vcops.controller.BaseExecutor.executeWithPagination - getCollectors failed: Federation call is not implemented.
 

Note: The above message is observed after the Analytics startup complete message

Environment

VMware Aria Operations 8.x

VMware vRealize Operations 8.x

VMware vRealize Operations Manager 7.0.x
VMware vRealize Operations Manager 7.5.x

Cause

This issue is caused by a misconfiguration of the $ALIVE_BASE/user/conf/cis.properties file.
The MY_LDUID property must have the same value across all nodes in the vRealize Operations cluster.

Resolution

This requires database intervention and it is advised to contact Broadcom Support  over a SR ticket for further investigation