Aria Operations cluster stuck in "Waiting for Analytics" due to high network latency
Article ID: 448413
Updated On:
Products
Issue/Introduction
- Aria Operations clusters configured in High Availability modes may experience intermittent service degradation.
- The admin interface reports the analytics nodes' state as Waiting for Analytics
- The primary Aria Operations web user interface becomes inaccessible.
- Data collection from adapter instances stops, and farms report "Adapter instance is not receiving data" alerts.
- The analytics service is terminated via SIGKILL after a 120-second shutdown timeout.
- analytics-wrapper.log repeatedly reports
ERROR: Shutdown failed: Timed out waiting for signal from JVM. In gemfire-locator.log we see below
Client Queue Initialization Threads stuck in TIMED_WAITING for CreateRegionReplyMessage- Network latency between the nodes is beyond 5 ms.
- The nodes in the cluster are across different physical location.
Environment
VMware Aria Operations 8.18.x
Cause
The root cause is network latency beyond the supported requirement. Aria Operations analytics nodes within a cluster require a round-trip time (RTT) latency of less than 5 ms. When latency exceeds this threshold, GemFire communication deadlocks occur. These deadlocks prevent subscription queue cleanups and cause the analytics engine to hang or crash during metadata synchronization.
Resolution
To resolve this issue, the cluster must be aligned with supported latency requirements. Refer: VMware Aria Operations 8.18 Sizing Guidelines
Additional Information
All analytics nodes for a single Aria Operations cluster must be placed in the same data center. The documented network requirement for analytics-to-analytics communication is < 5 ms latency.
Feedback
