VCF Operations cluster nodes stuck in "Waiting for Analytics" status

Products

• VCF Operations

Issue/Introduction

• Primary and or Replica nodes are stuck in a "Waiting for Analytics" status within the Admin UI.
• The Data node may show as "Online", but integrations are not collecting data.
• "CPU stuck/hung" messages may appear on the console of affected nodes.
• The VCF Operations product UI is accessible, but displays no live data or indicates cluster degradation.

Environment

VCF Operations 9.x

Cause

The issue is caused by intermittent inter-node communication or GemFire heartbeat failures.

Resolution

1. Log in to the vSphere Client and take snapshots of all VCF Operations nodes.
2. Log in to the VCF Operations Admin UI.
3. If the cluster status is stuck in "Going Online" or "Waiting for Analytics" select "Take Cluster Offline".
4. If the standard offline task fails, select "Force Take Cluster Offline".
5. Once the cluster is offline, Navigate to the vCenter Server and locate the VCF Operations VMs.
6. Perform a "Restart Guest OS" on each appliance in the cluster sequentially.
7. Return to the admin UI and select "Bring Cluster Online"