Root file system inside the TKC runs out of space because the log rotation isn't working.

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

The "/var/log/messages" inside the Guest Cluster nodes fills up the root partition causing the nodes to go unhealthy. In the worst-case scenario, this might lead to pods getting evicted out of the nodes because of the disk pressure caused by the lack of space.

Environment

This is expected in any Tanzu Kubernetes Cluster with TKR version less than v1.32.

Cause

Logrotate isn't added to the TKRs' as a part of the OVA generation process.

Resolution

Issue has been fixed in TKr v1.32.0 for vSphere 8.x

Workaround:

1. Using Logrotate: Validate logrotate is installed and available in the system. If not, install using below commands.

   •   For Photon: sudo tdnf install logrotate -y
       For Ubuntu: sudo apt install logrotate -y
     
     

     By default, the entry should be available in /etc/cron.daily/logrotate. If not, then update the `/etc/logrotate.d/messages` to include the below configuration.

     
     /var/log/messages {
    compress
    nodateext
    rotate 5
    daily
    maxsize 100M
    missingok
    notifempty
    copytruncate
}

     This will cause the /var/log/messages to get rotated either when the size crosses 100M or daily. For this to complete, the logrotate command should run successfully. 
     By default, the entry inside /etc/cron.daily/logrotate should run the daily logrotate command which in-turn should rotate the logs. Please update the value in `maxsize` depending upon the disk space availability.

     If for some reason, we need to manually rotate /var/log/messages, run the below command.
     
          sudo /usr/sbin/logrotate /etc/logrotate.d/messages 

     The above workaround is applicable to versions earlier than TKr v1.32. Starting with TKr v1.32.0, a fix has been introduced that enables daily log rotation by default.

   • However, in certain scenarios where daily log rotation may not be sufficient, the rotation frequency can be increased to hourly by implementing the following additional configuration.
     
     Create or update the file /etc/systemd/system/logrotate.timer.d/override.conf with the following content:
     
     [Unit]
Description=Hourly rotation of log files
Documentation=man:logrotate(8) man:logrotate.conf(5)

[Timer]
OnCalendar=hourly
RandomizedDelaySec=1h
Persistent=true

[Install]
WantedBy=timers.target
     
     After updating the configuration file, restart the logrotate timer service and reload the systemd daemon using the following commands:
     
     systemctl restart logrotate.timer
systemctl daemon-reload
     
     A permanent fix that increases the log rotation frequency is available starting from minor version 1.32.10, 1.33.6, 1.34.2,1.35.

2. Mounting /var/log/messages: mount /var/log/messages to a separate volume with enough disk space.