VCFA: Very large log bundles are created on VCF Automation 9.0.0, even when just one day is selected.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Excessively large VCF-A log file sizes causing issues when creating log bundles and attaching to support cases
• In a greenfield or brownfield installation of VCFA 9.0.0.0, log bundles for a single day are tens of gigabytes in size
• /var/log/messages may be many GB by itself
• /var/log/services-logs directory may be more than 10GB

Environment

VMware Cloud Foundation Automation 9.0.0

Cause

Prelude logs that are written to the node are supposed to get truncated if the node disk usage exceeds predefined limit. However, that cleanup code had a bug in 9.0.0 which was fixed in 9.0.1, which prevented it from running to completion in some circumstances.

Resolution

The issue is resolved in VCFA 9.0.1

 

Workaround

Before proceeding, please ensure you have a valid backup or snapshot in case needed logs are being cleared in the below procedure.

Basic things that need to be checked:

1. Available storage on each node. If it exceeds 75%, delete the content of /var/log/vmsp-logs, /var/log/services-logs and /var/log/messages

   • rm -rf /var/log/vmsp-logs/* /var/log/services-logs/*
   • truncate -s 1M /var/log/messages

2. Check if all nodes are ready. It might take a few minutes for nodes to show up as ready after disk space cleanup. A node replacement might get initiated, check for that.

   • kubectl get machines -n vmsp-platform

3. Check if pods in namespaces kube-system and vmsp-platform are running and not in restart loop.

   • kubectl get pods -n vmsp-platform
   • kubectl get pods -n kube-system

4. Check if the object storage and all buckets are not full. This can be done using this command on any node
   (need to set KUBECONFIG variable first):

   • export KUBECONFIG=/etc/kubernetes/admin.conf
   • kubectl exec -i -n vmsp-platform seaweedfs-master-0 -- weed shell 2>/dev/null <<< 's3.bucket.list'

If utilization of support-bundle bucket is high (>85%), it would need to be cleared. First, generate support bundle to preserve current logs (if haven't done already), then do this:

1. kubectl exec -it -n vmsp-platform seaweedfs-master-0 -- weed shell
2. fs.rm -r /buckets/support-bundle/
3. fs.cd /buckets/support-bundle
4. fs.rm -r system-logs services-logs events audit apiserver-audit