[9.1.1] VCF Management Services Node Resource Usage Approaching Capacity

Products

• VCF Operations

Issue/Introduction

The following alert appears in VCF Operations:

• VCF Management Services Node Resource Usage Approaching Capacity.
• A node in the VCF services runtime cluster is running low on available disk space. 

Depending on which component is impacted, the following behaviour may also be observed:

• VCF services runtime
  • Issues with deploying new components, patch/upgrade and configure actions might fail.
• Software depot
  • Issues downloading product binaries for install, patch/upgrades.
• Identity Broker
  • SSO/Login functionality might be impacted.
• Log Management
  • Issues with pushing logs to Ops.

 

Environment

• VCF Operations 9.1.1
• VCF Services Runtime 9.1.1

Cause

Disk pressure on a VCF management services node is usually caused by growth in one or more of the following:

• Service logs (/var/log/services-logs) growing faster than they're rotated and forwarded — typically from a high volume of lifecycle operations (install, patch, upgrade, configure) or API calls against VCF management services components.
• Audit logs (/var/log/audit, /var/log/apiserver-audit) growing due to a high rate of Kubernetes API server or Linux audit events — often from frequent lifecycle operations or automation/monitoring tools polling the API.
• Container images (/var/lib/containerd) accumulating from repeated install/patch/upgrade operations — each lifecycle task pulls a new component image, and old images aren't always cleaned up automatically once the task completes.
• Resource profile capacity — the node is functioning correctly but is hosting more concurrently deployed/managed components than its current disk sizing supports.

Resolution

1. Confirm the Affected Node
   1. In VCF Operations, navigate to Operate > VCF Health > Component View > VCF Management Services.
   2. In the Nodes Health section, identify:
      1. The name of the affected node and it's disk usage
      2. Whether the alert appeared suddenly or has been building gradually.
2. Check for Active Lifecycle Operations
   1. Navigate to Build > Lifecycle > Tasks.
   2. If any task (upgrade, patch, configure, or certificate rotation) is in Running state, wait for it to finish. Disk usage from component image downloads and temporary workspace data typically returns to normal within 30–60 minutes after the task completes.
   3. If no tasks are running and disk pressure is sustained, proceed to Step 3.
3. Use Nodes Health section under VCF management services health dashboard (Operate > VCF Health > Component View > VCF Management Services) to check the resource usage of the node.
4. Establish an SSH session to the node.
   1. https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/fleet-management/configuring-management-components/enable-ssh-access-to-the-vcf-services-runtime-cluster.html
5. Validate current disk usage
   1. Overall filesystem usage
      1. df -h
   2. Size of the service-logs directory
      1. du -sh /var/log/services-logs
   3. Size of archived (.gz) files in service-logs
      1. du -ch /var/log/services-log
6. Remove potential storage bottlenecks
   1. Archived files in service-logs (already forwarded to support bundle, safe to delete)
      1. find /var/log/services-logs -name "*.gz" -type f -delete
   2. Other rotated log files in service-logs (skips the active, currently-written log file)
      1. find /var/log/services-logs -type f ! -name "*.log" -delete
   3. Rotated audit logs (active log is kept; apiserver-audit only exists on control-plane nodes)
      1. find /var/log/audit -name "audit.log.[0-9]*" -type f -delete
      2. find /var/log/apiserver-audit -name "kube-apiserver-audit.log.*" -type f -delete 2>/dev/null
   4. Unused container images (only removes images not in use by a running container)
      1. crictl rmi --prune
7. Once the files have been removed, confirm the storage has been reclaimed
   1. df -h