Requested redundancy is incompatible with existing redundancy in Aria Operations

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Implement the resolution in the KB only when you see the errors below reported in the logs:

• The vRealize Operations Manager cluster is stuck in a Waiting for Analytics state.
• In the /storage/log/vcops/log/analytics-<UUID>.log file on one or more nodes, you see the errors:
  

  ERROR [Analytics Main Thread ] com.vmware.vcops.controller.gemfire.GemfireCommunicator.connect - Cannot connect to gemfire. The reason is Requested redundancy 0 is incompatible with existing redundancy 1
  ERROR [Analytics Main Thread ] com.vmware.vcops.controller.gemfire.GemfireCommunicator.connect - Cannot connect to gemfire. The reason is Requested redundancy 1 is incompatible with existing redundancy 0

Environment

VMware Aria Operations 8.x
VMware vRealize Operations 8.x 

Cause

This issue occurs when one or more nodes has the Shard Redundancy level set incorrectly from the rest of the nodes in the cluster.

Resolution

Mandatory: Take snapshots - Snapshot Creation in VMware Aria Operations
Snapshots are required on all analytics (Primary, Replica, and Data) nodes in the cluster before following the steps below. It is not necessary to take snapshots of Cloud Proxies, as Cloud Proxies are not members of analytics cluster.

To resolve this issue, the Shard Redundancy level should to be corrected on the node(s) with the correct value.

To correct the Shard Redundancy level:

1. Log in to the primary node as root through console or SSH.
2. Stop all VMware Aria Operations services by running this command:
   

   service vmware-vcops stop

3. Open the /usr/lib/vmware-vcops/user/conf/gemfire.properties file using a text editor.
4. Change the line containing 'shardRedundancyLevel' to 1 (enabled) or 0 (disabled), depending on the state of HA.
   
   Note: The state of HA can be verified in the VMware Aria Operations admin UI of any node.
   
   
5. Save and close the file.
6. Repeat steps 1 to 5 on all analytics nodes in the Aria Operations cluster.
7. Start all VMware Aria Operations services at the same time on all the nodes by running this command. Starting with Primary, then Replica, and finally all Data nodes
   

   service vmware-vcops start

Note: Shard Redundancy refers to High Availability in Aria Operations. Valid states for the redundancy level are 1, HA enabled, or 0, HA disabled.