Aria Operations nodes crash or show "Waiting for Analytics" due to Java Heap Exhaustion from MS SQL Event Flood

Products

• VCF Operations

Issue/Introduction

• One or more Aria Operations nodes are unresponsive or in a "Waiting for Analytics" state.
• Cluster Status in the Admin UI shows Degraded.
• Following error is observed in /storage/vcops/log/analytics-wrapper.log or adapter logs: java.lang.OutOfMemoryError: Java heap space
• Large heap dump files (.hprof) are present in /storage/db/vcops/heapdump/.
• Environment has the Management Pack for Microsoft SQL Server configured.

Environment

Aria Operation 8.18

Cause

This issue occurs when a monitored Microsoft SQL Server generates a massive volume of repetitive error logs (e.g., due to a disk-full condition on the SQL server itself).

The Microsoft SQL Server adapter queries xp_readerrorlog every collection cycle. While it deduplicates messages within a single 5-minute cycle, it has no cross-cycle memory. If the underlying SQL issue persists for days, the adapter continues to ingest and forward the same events in every cycle. This leads to a massive accumulation of event objects in the Aria Operations analytics JVM heap, eventually causing a crash.

Resolution

1. Deactivate log-based event collection on the affected adapter instance to stop the memory pressure immediately.

In the adapter instance configuration, under Advanced Settings, set Exclude Log Based Events = True. This stops the adapter from querying xp_readerrorlog entirely on the next collection cycle. The event flood stops immediately. Other event types (index events, file capacity events, job failure events) are unaffected.

2. Configure a safety cap to limit the number of events the adapter can process per cycle, preventing a recurrence of heap exhaustion.

• SSH into the Aria Operations node (or Cloud Proxy) where the adapter is running.
• Edit the adapter properties file: vi /usr/lib/vmware-vcops/user/plugins/inbound/sql_server_adapter/conf/microsoftsqlserver.properties
• Add or update the following property (setting it to 100 or another reasonable limit): maximum_number_of_events_to_return_per_collection=100

3. Free up the disk space on the SQL server . Once the disk issue is resolved, the log entries stop.