Tanzu Kubernetes Grid Control Plane upgrade fails due to journal log disk usage limits

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• During a Tanzu Kubernetes Grid (TKG) or vSphere with Tanzu (VKS) upgrade, the process fails when attempting to push new images to the control plane nodes. Pre-checks or upgrade tasks indicate that the file content on the control plane exceeds allocated disk limits.
• This prevents the lifecycle management process from staging the necessary upgrade artifacts.

Environment

TKGm 2.5.x

Cause

Control plane node disk sizing constraints. High journal log accumulation on the control plane nodes consumes the available disk space required for upgrade image staging.

Resolution

Identify and clear excessive log data to bring disk usage below the required threshold before re-initiating the upgrade.

1. Access the affected control plane node via SSH.
2. Verify the current disk usage of the systemd journal logs: 
   journalctl --disk-usage
3. If usage is excessive, rotate and prune the journal logs to a maximum size (e.g., 500MB): 
   journalctl --vacuum-size=500M
4. Restart the logging service to apply changes: 
   systemctl restart systemd-journald
5. Verify disk availability and retry the upgrade.