Resolving Disk Space (/storage/var/log) Issues in Aria Operations for Logs: QueryResultsCache Growth & Upgrade Failures

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• /storage/var/log runs out of disk space.
• /storage/var/loginsight/cache/QueryResultsCache.data is several GB in size.
• If /storage/var/log becomes full, the UI might not be accessible.
• While trying to upgrade Aria Operations for Logs, the upgrade will not proceed as the /storage/var partition is full.
• The following error/warning can be found in the "upgrade.log" located in "/storage/core/loginsight/var".
  

  Not enough disk space! Available disk space at /storage/var: {Total disk space available} is less than required disk space: {Total requires disk space}

Environment

Aria Operations for Logs 8.x

Cause

The QueryResultsCache.data file in the "/storage/var/loginsight/cache" directory is growing disproportionately in size and consuming significant disk space.

The same can be verified by running the following command:

cd /
du -h | sort -h

Resolution

The following workaround can be applied:

• Take a SNAPSHOT of all the Aria Operations for Logs nodes.
• Stop the Log Insight service:

  service loginsight stop

• Change the directory using the following command:

  cd /storage/var/loginsight/cache

• List the files using the following command to ensure the file really exists:

  ls -lh

• Delete "QueryResultsCache.data" using the following command. If not successful, delete both "QueryResultsCache.lock" and "QueryResultsCache.data".

  rm QueryResultsCache.data
  rm QueryResultsCache.lock

• Start the Log Insight service:

  service loginsight start

Additional Information

Impact:

• Deleting the cache will not affect system functionality, but the cache file will be recreated when new queries are made.
• This resolution may help free up space in the /storage/var/log directory, allowing the system to operate correctly again.

Also Refer: How to safely clean /storage/var space in Aria Operations for Logs (Formerly vRealize Log Insight).