/storage/core become nearly full due to size of buckets

Products

• VMware vRealize Log Insight 8.x
• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Running df -h command shows that the /storage/core disk usage is over 97%.
  
  
• Running du -h /storage/core/loginsight/cidata/store/ | sort -rh command shows buckets (/storage/core/loginsight/cidata/store/####) with a size greater than 500 MB:
  

  # du -h /storage/core/loginsight/cidata/store/ | sort -rh
  110G /storage/core/loginsight/cidata/store
  8.0G /storage/core/loginsight/cidata/store/####
  8.0G /storage/core/loginsight/cidata/store/####

  
  
  
• In /storage/core/loginsight/var/runtime.log, there are no recent activity log for "BucketManager-thread-1".

Environment

VMware Aria Operations for Logs 8.18.x

Cause

Because no timeout is configured for the name resolution check process, if "host -N" command executed by BucketManager does not return a response, BucketManager gets stuck.
Consequently, the BucketManager process stops managing the lifecycle of buckets, the bucket status remains unchanged, and its size continues to grow.

Resolution

Workaround

Restarting the loginsight service resolves the issue temporarily:
# systemctl restart loginsight

Resolution

To resolve the issue, execute the following commands to check the name resolution in the environment:

# host -N 0 <hostname of the Aria Operations for Logs node>
# nslookup <hostname of the Aria Operations for Logs node>
# dig <hostname of the Aria Operations for Logs node>

If the output does not return a successful response, verify the configuration of the DNS server and its environment.

Additional Information

Modifying the DNS settings of Aria Operations for Logs