Aria operations nodes with Split Brain scenario

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Primary as well as Replica sees themselves as Primary
  https://knowledge.broadcom.com/external/article/428875/emergency-cluster-bypass-for-vmware-iden.html
  
• However, the Primary and Replica shows correctly under repl.db.role
  https://knowledge.broadcom.com/external/article?articleNumber=414646
• Broken replication across the link of Primary and Replica causing dispensaries.

Environment

Aria Operations 8.17x

Cause

Average Network Latency will be higher than normal, but not disastrous. Hence restarting the Aria Ops cluster will intermittently suppress the issue.

• Replica with repeated events on reports  `Utils.checkConnection - TimeoutException` under  `Replica/logs/hafailover-<uuid>.log`
  Replica with repeated events on reports `master db seems down, double check.` under `Replica/logs/hafailover-<uuid>.log` |
• Replica starts streaming WAL from primary events observed in Replica/logs/postgres/repl/postgresql-xx.log
• Primary with repeated events on reports NoRouteToHostException toward Replica, and Connection refused causing loose connectivity, and stops logging.

  casa.log:

  YYYY-MM-DDTHH:MM:SS,331+0000  WARN [ajp-nio-127.0.0.1-8011-exec-26] [sQ0000Ne] casa.view.ClusterViewService:163 - Node is unreachable: <IP_Replica>. com.vmware.vcops.casa.exception.HttpMapFunctionUnreachableHostException: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<IP_Replica>/casa/view/slice": No route to host (Host unreachable); nested exception is java.net.NoRouteToHostException: No route to host (Host unreachable)

• Causing Replica locator restarts under  Replica/logs/gemfire-locator.log

• Replica management service resumes logging as Primary events under Replica/logs/casa-webapp/casa.log 

Resolution

Customer must first engage with Internal Network team to analyze and fix the Network latency.

Broadcom Recommendations;

• HA cannot handle this. CA with a witness can.

A two-node High Availability cluster has no way to resolve a partition. When the link fails, each node sees the same thing: my peer is unreachable, I am healthy. Both reach the only conclusion available to them, and both are right from where they stand. The information needed to decide is not present in a two-node system, so no amount of tuning changes the outcome. Enabling HA on this cluster would not have prevented the data loss. HA provides redundancy — a second copy, a second place to run. It does not provide arbitration.

 

• Continuous Availability adds the missing element — a witness node in a third failure domain — which turns an unresolvable two-way disagreement into a majority decision. During such events as in the Cause Analysis:

1. CA detects the suspected partition;
2. tests reachability in both directions and consults the witness;
3. establishes which side holds the majority, the witness being reachable from at most one of them;
4. actively shuts down the minority side — fencing it, not asking it to stand down.

 

• Adding a Witness node in a separate Network Domain.

  The Witness Node is the third network domain that exists independently and identifies network partitioning across the two fault domains. If network connectivity between the two fault domains is lost, the cluster goes into a split-brain situation, which is detected by the Witness Node. Immediately, one of the fault domains goes offline to avoid data inconsistency problems.

  
  Continuous availability for VMware Aria Operations 

https://techdocs.broadcom.com/us/en/vmware-cis/aria/aria-suite-lifecycle/8-18/vmware-aria-suite-lifecycle-installation-upgrade-and-management-8-18/creating-a-vidm-environment/create-a-new-private-cloud-environment-using-an-install-wizard/configure-vrealize-suite-products-for-installation/continuous-availability-for-vrealize-operations-manager.html

Additional Information

Additionally, Logs as "I/O error on GET request for "https://<IP_Replica>/casa/view/slice": No route to host (Host unreachable)" in the casa logs can be observed when IP routable errors observed as in, https://knowledge.broadcom.com/external/article/412109/unable-to-log-in-to-the-aria-operations.html

Which also advised to customer engaging with internal network team.