VCF Operations for Logs UI Performance Degradation and Primary Node CPU Spikes

Products

• VCF Operations

Issue/Introduction

Users may observe the following issues after deploying or updating VCF Operations for Logs (formerly Aria Operations for Logs):

• The Web UI is slow, unresponsive, or fails to load dashboards.
• High CPU utilization (spiking to maximum capacity) on the primary cluster node.
• Log collection pages appear stuck with numerous pending tasks.
• Cluster information pages fail to populate.

Environment

VCF Operations for Logs 9.0.x

Cause

The /storage/var partition on the primary node has reached 100% utilization. This is typically caused by the accumulation of large heapdump (.hprof) files generated when system services encounter an exception or crash. Because these files can be several gigabytes in size, they quickly exhaust the limited space in the /storage/var partition, leading to system instability.

Resolution

To reclaim disk space and restore UI performance, follow these steps to remove the heapdump files and reboot the affected node.

Prerequisites

Take a snapshot of all nodes in the VCF Operations for Logs cluster before performing manual file deletions.

Procedure

1. Log in to the Primary Node: Access the node as root via SSH or the vSphere console.
2. Verify Disk Space: Confirm the partition usage by running:

   df -h /storage/var

   If the Use% is 100%, proceed to the next step.
3. Check for Heapdump Files: Identify the size and location of .hprof files:

   du -hd1 /storage/var/loginsight/heapdump/
   ls -l /storage/var/loginsight/heapdump/*.hprof

4. Remove the Files: Delete the heapdump files to free up space:

   rm /storage/var/loginsight/heapdump/*.hprof

5. Reboot the Node: Perform a forced reboot to restart all services cleanly:

   reboot -f

6. (Optional) Verify Load Distribution: Ensure log ingestion is balanced across the cluster under the Management -> System Monitor -> Statistics page.
   If vCenters are targeting a single node's IP, reconfigure them to use the Load Balancer Virtual IP (VIP) instead from the Log collection page in the VCF Operations UI.