Troubleshooting "Waiting for Analytics" in Aria Operations

Products

• VMware Aria Operations (formerly vRealize Operations) 8.x

Issue/Introduction

One or more nodes in an Aria Operations cluster remain stuck on:

"Waiting for Analytics"

This prevents the cluster from reaching a fully operational state.  The issue may occur after a reboot, outage, upgrade, or environmental change.

Note: It is normal for the nodes to be "Waiting for Analytics" for 5 - 10 minutes during start up.

Environment

Aria Operations 8.x and later

Cause

The "Waiting for Analytics" state is a transitional condition that indicates the analytics engine has not successfully initialized.  Common root causes include:

• Name resolution failures (DNS misconfigurations)
• Storage or I/O saturation or full partitions
• Time synchronization drift (NTP)
• Memory exhaustion
• Start up order
• Continuous Availability replication, latency, or start up issues

Resolution

These articles are listed in the order of the most likely cause.  Start with the Tier 1 articles (Core infrastructure, Configuration Issues, and Cluster Integrity) first, and move to the Tier 2 articles (Continuous Availability if in use and Upgrade Issues if this is happening during an Upgrade) second.  Tier 1 issues could be affecting a Continuous Availability cluster as well as a cluster being Upgraded.  

Tier 1

Core Infrastructure

KB	Title	Summary
398765	Restarting all nodes after bringing the cluster Offline , the cluster status stays on "Going Online" and nodes show "Waiting for analytics..." for more than an hour	Correct vApp Properties for DNS servers
318408	Troubleshooting Storage Issues in Aria Operations	Check Space on /, /storage/db, and /data
315903	Aria Operations 8.x Cluster fails to start with the status "Waiting for Analytics".	Time drift issues, check NTP

Configuration Issues

KB	Title	Summary
378322	Aria Operations node stuck on "Waiting for Analytics" status	Manual modification of the vPostgres certificate is not supported.
403206	Analytics down - Aria Operations	Monitoring Kubernetes guest filesystems causes java heap crash dumps

Cluster Integrity

KB	Title	Summary
372809	Aria Operations cluster status stuck on Waiting for Analytics, analytics-.log has the error "Attempt to start not runnable node"	Analytics processes are not able to come online due to the divergence of the "CACHED_ROLES" document between the nodes.

Tier 2

Continuous Availability

KB	Title	Summary
368819	Continuous Availability Cluster displaying: "Waiting for Analytics" with error: "Attempt to start node from not accessible zone"	Fault Domain is marked as OFFLINE/FAILURE in casa
402973	Primary replica node status changes to "Waiting for analytics" while the cluster state becomes "Degraded"	Continuous Availability has very strict latency and packet loss threshold requirements. If these thresholds are breached consistently, Continuous Availability will not function properly.
406434	Aria Operations cluster data collection stops functioning intermittently in Continuous Availability environment.	Network latency between fault domains should ideally be less than 10 ms, with occasional peaks allowed up to 20 ms during 20-second intervals.
406917	Data center went down, attempting to bring the cluster online, but nodes stuck at "waiting for analytics" after everything is up	Power on order

Upgrade Issues

KB	Title	Summary
379482	Upgrading Aria Operations Manager hangs. The process is stuck at step 10 of 14 "Applied product update" due to CustomRestPlugin.	This issue occurs when an unsupported plug-in is installed. If there are any "CustomRestPlugin" being used, it leads to the upgrade failure.
391506	Unable to start upgrade. All slices should be in the same state (online or offline) when installing a management pack	Power on order