Why a 1-Locator, 2-Server Configuration is Not Recommended for Tanzu GemFire Production Environments

Products

• VMware Tanzu Data Suite

Issue/Introduction

In VMware Tanzu GemFire, deploying a cluster configuration consisting of only one locator and two cache servers is highly discouraged for production environments. This specific topology is vulnerable to quorum loss and cannot reliably handle a single member failure without risking a complete cluster shutdown.

Why This Configuration is Risky GemFire uses a member weighting system to manage network partitions and prevent split-brain scenarios. By default, the enable-network-partition-detection property is set to true to prevent data divergence. In this mode, if the system loses more than 50% of its total member weight, the surviving members on the "losing" side will shut themselves down to protect data integrity.

Member weights are typically assigned as follows:

• Locator: 3

• Cache Server: 10

• Lead Cache Server (Oldest living server): 15

The Quorum Math for 1 Locator and 2 Servers:

• Total Cluster Weight: 3 (Locator) + 15 (Lead Server) + 10 (Server 2) = 28

• Quorum Threshold: 50% = 14

The Failure Scenario: If the host running the Lead Server (weight 15) fails, the remaining members (the locator and the second server) only possess a combined weight of 13. Since 13 is strictly less than the 50% quorum threshold of 14, the remaining locator and server will automatically shut themselves down, resulting in a total cluster outage.

Resolution

To ensure high availability and prevent total cluster failure due to quorum loss, you must adjust the cluster topology. Implement the following recommendations:

1. Deploy a Minimum of 3 Cache Servers To maintain quorum during a single server failure, deploy at least three cache servers. This ensures that losing any one server (even the heavily weighted Lead Server) does not cause the remaining cluster weight to drop below the 50% threshold.

2. Deploy Redundant Locators Run at least two locators on separate hosts. This ensures that the loss of a single locator does not leave the cluster without its discovery mechanism and prevents a single point of failure for client connections.

3. Do Not Disable Partition Detection Broadcom highly recommends leaving enable-network-partition-detection set to true. While disabling it might keep the cluster running during a split, it can lead to a "split-brain" scenario where two halves of the cluster operate independently, resulting in unrecoverable data divergence.