Guest Cluster API Unreachable and control plane Node NotReady Due to Full Root Partition

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Guest Cluster API is unreachable, and the cluster status shows Ready: False.
• The containerd service fails with the error: error fail to write log to log file /var/log/pods/xxx
• The root (/) partition on the control plane VM is 100% full.
• A large file (e.g., 11GB) is identified at /var/log/messages

Environment

Tanzu Kubernetes Cluster v1.30.1

Cause

This issue occurs when system logs, specifically /var/log/messages, grow unbounded and consume all available space on the root partition.

In Tanzu Kubernetes Grid (TKG) versions earlier than v1.32, log rotation might not be configured by default, leading to disk pressure that evicts pods and prevents the container runtime from writing logs.

Resolution

To restore cluster operations immediately, truncate the bloated log file to free up space:

truncate -s 0 /var/log/messages

Additional Information

Upgrade to Tanzu Kubernetes Release (TKr) v1.32.0 or later. Starting with this version, daily log rotation is enabled by default to prevent disk pressure issues.
https://knowledge.broadcom.com/external/article?articleNumber=383366