CA cluster deployment stuck at starting the cluster - VCF Operations

Products

• VCF Operations

Issue/Introduction

When deploying a new VCF Operations cluster or expanding an existing one with High Availability (HA) or Continuous Availability (CA), you may experience the following symptoms:

• The cluster status in the Administration UI remains stuck at Starting the Cluster during initial installation.
• Individual nodes (Primary, Replica, or Data nodes) display a status of Waiting for Analytics
• Analytics logs (/storage/vcops/log/analytics-<UUID>.log) show errors such as: com.integrien.analytics.AnalyticsMain.createGemfireCache - Can not connect to gemfire: Problem starting up membership services. 

Environment

VCF Operations 9.x

Cause

This issue is typically caused by incomplete network connectivity between the cluster nodes. VCF Operations requires specific internal ports to be open for GemFire membership services and data synchronization, even if the nodes reside on different VLANs or subnets

Resolution

Ensure the network environment and appliance configuration meet the following requirements:

1. Verify Port Connectivity

Work with your network team to ensure that the following ports are open between the nodes in the cluster (Note: Primary / Replica nodes are also classified as Data Nodes):

 

 

Source Node	Destination Node	Port(s)	Usage
Data Node	Primary / Replica	TCP 6061	Communication with Geode Locator
Data Node	Primary / Replica	TCP 20002 - 20010	Geode TCP inter-node failure detection & peer-to-peer communication
Data Node	Primary / Replica	TCP 5433	Communication with Postgres Central DB
Data Node	Data Node	TCP 443	HTTPS
Data Node	Data Node	TCP 10000	Communication with Geode server embedded in Analytics process
Data Node	Data Node	TCP 10002 - 10010	Geode TCP inter-node failure detection & peer-to-peer communication
Data Node	Witness Node	TCP 443	HTTPS
Witness Node	Data Node	TCP 443	HTTPS

2. Validate with Netcheck Utility

Run the internal Netcheck.py script from the appliance CLI to identify specific connection failures:

/usr/bin/python /usr/lib/vmware-vcopssuite/python/lib/Netcheck.py

If any ports return a FAILED status, the firewall or routing configuration must be adjusted.

3. Configure DNS and Reverse DNS

Ensure that all nodes have forward (A) and reverse (PTR) records configured in your DNS server. Missing PTR records often lead to the Waiting for Analytics state as nodes fail to resolve each other during the join process.

4. Cross-Subnet Routing

If nodes are deployed across different IP networks or VLANs:

• Confirm that latency between nodes is within supported limits (typically < 5ms/10ms for HA/CA).
• Ensure that MTU settings are consistent across the network path to prevent packet fragmentation.