Continuous Availability cluster fails to come online due to 100% /storage/log utilization on Witness node in VMware Aria Operations

Products

• VCF Operations

Issue/Introduction

• In VMware Aria Operations configured with a Continuous Availability (CA) architecture, the cluster fails to initialize and come online following a graceful shutdown and restart.
• The Witness node remains down, preventing the cluster to come online.

Environment

Aria Opertaions 8.18.5

Cause

This is primarily caused by unrotated localhost_access logs accumulating within the /storage/log/vcops/log/CASA directory, which eventually exhausts all available space on the partition and prevents required services from starting.

Resolution

To resolve this issue, manually clear the stale log files from the Witness node to reclaim disk space.

1. Establish an SSH session to the Aria Operations Witness node and log in as root.

2. Verify the disk space utilization of the partitions by running: df -h

3. Navigate to the CASA log directory: 

   cd /storage/log/vcops/log/casa

4. List the files to confirm the presence of large or numerous localhost_access logs: 

   ls -lhrt

5. Delete the old and unrotated localhost_access log files. (Adjust the wildcard pattern as necessary for your environment's stale files):

   rm localhost_access_log.*.txt

6. (Optional) If additional space is required, review and clear older audit or messages logs within the /var/log directory.

7. Re-run df -h to confirm the /storage/log partition utilization has been significantly reduced (e.g., below 80%).

8. Proceed to the Aria Operations Admin UI and bring the CA cluster online.