Root "/" full on worker node(s)

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• System pods crashing due to control plane nodes with disk pressure due
• Users unable to mount volumes as the CSI controller pods are failing
• Worker nodes have root file system full at 100%
•  /var/log/messages is taking huge size in gigabytes filling the / dir

Environment

2403.2.15

Cause

• The /var/log/messages (Photon OS) and the /var/log/syslog (Ubuntu) directories serve as the central system log repo within a link system. 
• The logs files within these directories are not rotating or being cleaned up as expected.

Resolution

Clean up the central system log repositories

Note: Photon OS uses /var/log/messages while Ubuntu uses/var/log/syslog. 

1. Switch to the root user:
   su -
2. Truncate the messages or syslog directory, based on the OS in use:
   truncate -s 0 /var/log/messages
truncate -s 0 /var/log/syslog
3. Remove compressed log files:
   rm -rf /var/log/messages.*.gz
rm -rf /var/log/syslog.*.gz

Additional Information

You can also vacuum the Journal logs:

Review Disk usage:
journalctl --disk-usage

Vacuum the Journal logs:
journalctl --vacuum-size=100M