One or more nodes in an Aria Operations cluster remain stuck on: "Waiting for Analytics"

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

One or multiple nodes of Aria Operations are stuck at "waiting for analytics" status.

Environment

Aria Operation 8.18.x

Cause

The "Waiting for Analytics" state is a transitional condition that indicates the analytics engine has not successfully initialized.  Common root causes include:

• Old/Unwanted logs fill up on one or multiple nodes.
• The /storage/db partition on one or more nodes has become full due to retained old heap dump files.
• Leftover snapshots are present on the local disks on one or multiple nodes of Aria Operations.

Resolution

• Verify the /storage/log utilization and consider removing old/unwanted logs on the nodes using the command:

   find /storage/log/ -mount -type f -mtime +1 -exec echo {} \; -exec truncate -cs 0 {} \; 2>&1 | tee /tmp/files_truncated.txt

• Check for heapdump files from all the nodes using the command:

  find /storage/db -name \*.hprof -exec ls -lah {} \;
  

  Remove all old/unwanted heap dump files using the command:

  rm java_pid####.hprof

• Check for any old/stale snapshots on all aria operations node(s) disks.
• Delete the snapshot by following best practices for snapshot creation and removal for aria operations node(s) using the kb:
  Snapshot Creation in VMware Aria Operations (342576)

Additional Information

References:

• How to safely clean /storage/log in Aria Operations (315972)
• Heapdumps filling /storage/db in Aria Operations (317811)
• Managing snapshots in vSphere Web Client (344708)