vSphere Supervisor Control Plane VM Root Disk Space Full at 100%

Products

VMware vSphere Kubernetes Service Tanzu Kubernetes Runtime VMware vSphere 7.0 with Tanzu vSphere with Tanzu

Issue/Introduction

Root disk usage reaches 100% on one or more Supervisor Cluster Control Plane VMs in a vSphere Supervisor environment. This leads to disk space exhaustion in the root directory and disk pressure issues.
The following error message appears for one or more Supervisor control plane nodes when viewing the Config Status of the Supervisor under the Workload Management section of the vSphere Client:
System error occurred on Master node with identifier ################################. Details: Log forwarding sync update failed: Command '['/usr/bin/kubectl', '--kubeconfig', '/etc/kubernetes/admin.conf', 'get', 'configmap', 'fluentbit-config-system', '--namespace', 'vmware-system-logging', '--ignore-not-found=true', '-o', 'json']' returned non-zero exit status 1.
The Supervisor may show one node in error state.
The following error message appears when curling the kube-apiserver endpoint due to an etcd quorum loss:
curl --insecure https://<supervisor-floating-ip-address>:6443/healthz>>curl: (28) Failed to connect to <supervisor-floating-ip-address> port 6443 after 21051 ms: Could not connect to server
The following error appears in the journalctl output:
unable to register node with api server 6443 connection refused
While initiating an SSH connection to the Supervisor Control Plane VM, the root disk space displays 100%. Review the "How to SSH into Supervisor Control Plane VMs" section in the KB article Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VMs.
The floating IP (FIP) address output by the decryptK8Pwd python script is unreachable due to disk space issues bringing down critical system processes. Use the management network (eth0) IP address directly assigned to the Supervisor Control Plane VM instead of the floating IP address (FIP).
Verify the current disk space on the root partition:
df -h /dev/root
Filesystem Size Used Avail Use% Mounted on/dev/root ##G ##G ##G 100% /
Many system processes fail and crash when any Supervisor Control Plane VM is at full root disk usage. This includes the system service that assigns the floating IP address (FIP) to one of the Supervisor Control Plane VMs.

Environment

vSphere Supervisor

Cause

Disk usage on the cluster increases due to a variety of reasons:

Log Accumulation: /var/log
ETCD Snapshots and Data
Container/Pod Logs: /var/log/pods
Leftover unused images and replicasets built up over time from previous Supervisor cluster upgrades.

VMware by Broadcom Engineering is aware of the issue and includes fixes in vSphere 8.0 U3g, vSphere 9.0, and higher versions for the below known issues:

Failed log rotation of /var/log/vmware/upgrade-ctl-cli.log* files leading to multiple 1GB files appended with an additional number.
Unused images and replicasets build up over time and are left over from previous Supervisor cluster upgrades.
Excessive disk space usage populated by system journal logging and other logging system services.
Increased overall disk space of each Supervisor control plane VM.

Resolution

When the root disk space in a Supervisor control plane VM reaches 100%, multiple system-critical services fail. Clean up the root disk space on all Supervisor control plane VMs to restore the healthy operation of the Supervisor cluster. Maintain root disk space below 80%.

Warning: Deleting files without Support's advice leads to further issues or potential irrecoverable destruction of the environment.
Note: This KB article focuses primarily on log clean-up. Do not delete the latest log files; echo them empty if necessary.
For example, where example.log is a log file to be echoed empty:

echo > example.log

Steps for Resolution:

SSH into each of the Supervisor Control Plane VMs to execute the following steps. Review the "How to SSH into Supervisor Control Plane VMs" section in the KB article Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VMs.

Note: The floating IP (FIP) address output by the decryptK8Pwd python script is unreachable due to disk space issues bringing down critical system processes. Use the management network (eth0) IP address directly assigned to the Supervisor Control Plane VM instead of the floating IP address (FIP).

Perform these remediation steps for each Supervisor Control Plane VM in the Supervisor cluster:

Audit Logs - Affects all Supervisor Control Plane VMs
During downtime caused by disk space issues, the audit logs for kube-apiserver fill up rapidly with expected errors of the kube-apiserver being unreachable. The kube-apiserver is unreachable because disk space issues bring down critical system processes such as kube-apiserver and etcd.
ls -ltrh /var/log/vmware/audit

Clean up older .log.gz files in this directory.
Backup Files - Affects All Supervisor Control Plane VMs
The backup files involved in taking a vCenter backup in VAMI with the Supervisor Cluster included reside in this directory.
ls -ltrh /var/lib/vmware/wcp/backupClean up any older backup files in this directory.
upgrade-ctl-cli.logs - Affects all Supervisor Control Plane VMs
A known issue causes upgrade-ctl-cli.log log rotation to fail, repeatedly filling up multiple upgrade-ctl-cli.log files (appended with a number) up to 1GB each.
ls -ltrh /var/log/vmware/upgrade-ctl-cli*

Clean up older upgrade-ctl-cli.log.# files appended with a number to reduce disk space usage. Do not delete the latest upgrade-ctl.cli.log file.

Run the below command to limit the size of the log file to 10MB (IMPORTANT: Re-apply this sed command after any Supervisor upgrade until you apply the built-in fix):
sed -i '/MAX_LOGFILE_SIZE_BYTES =/ s/1024 * 1024 * 1024/1024 * 1024 * 10/' /usr/lib/vmware-wcp/upgrade/upgrade-ctl.py

Perform the extra upgrade-ctl-cli log.# files clean up and the above sed command on each Supervisor control plane VM in the Supervisor cluster.
journalctl logs - Affects all Supervisor Control Plane VMs
Journalctl logging affects disk usage. Compress the current journalctl logging to quickly free space.

To check the current disk usage for journalctl logging, run the below command:
journalctl --disk-usage

To compress and free up space, run the below command:
journalctl --vacuum-size=500M
Stale Replicasets and Stale Images - Affects all Supervisor Control Plane VMs
Note: This requires a healthy kube-apiserver and ETCD on all Supervisor Control Plane VMs. Perform the previous clean-up steps on each VM first.
These stale replicasets and stale images remain unused from previous Supervisor cluster upgrades.
Retrieve the total replicaset count in the Supervisor cluster (the expected healthy count remains under 60):
kubectl get replicasets -A | wc -l

Find duplicate stale images with different versions by checking the container images list while establishing an SSH session to a Supervisor control plane VM:
crictl images list

Review the vSphere Supervisor Disk Space Clean Up Scripts KB for scripts regarding cleaning up unused images and replicasets to help with disk space.
ETCD
Check the etcd database size to see if it has reached the 2GB capacity limit:
etcdctl member list -w table
etcdctl --cluster=true endpoint health -w table
etcdctl --cluster=true endpoint status -w table
ls -ltrh /var/lib/etcd/member/snap

If the above steps show that ETCD is 2GB or more, or if etcdctl does not return outputs or shows that at least one control plane VM is unhealthy, reach out to VMware by Broadcom Technical Support for assistance and reference this KB article. Supervisor cluster upgrades also reduce root disk space usage, but if the disk space fills back up, reach out to VMware by Broadcom Technical Support for assistance.

Additional Information

New versions of vSphere 9 and vSphere 8.0 include log rotation and disk space improvements.

To get assistance from a customer representative or a Support Engineer, check Contact Support.