Root file system inside the TKC runs out of space because the log rotation isn't working.

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

1. The "/var/log/messages" inside the Guest Cluster nodes fills up the root partition causing the nodes to go unhealthy.
2. In the worst-case scenario, this might lead to pods getting evicted out of the nodes because of the disk pressure caused by the lack of space.

Environment

This is expected in any Tanzu Kubernetes Cluster with TKR version less than v1.32.

Cause

Logrotate isn't added to the TKRs' as a part of the OVA generation process.

Resolution

Engineering is still working to address the same in the TKR versions v1.32 and later. Meanwhile the below workarounds are available.

1. Using Logrotate: Validate logrotate is installed and available in the system. If not, install using below commands.

    For Photon: sudo tdnf install logrotate -y
    For Ubuntu: sudo apt install logrotate -y

By default, the entry should be available in /etc/cron.daily/logrotate. If not, then update the `/etc/logrotate.d/messages` to include the below configuration.

/var/log/messages {
    compress
    nodateext
    rotate 5
    daily
    maxsize 100M
    missingok
    notifempty
    copytruncate
}

This will cause the /var/log/messages to get rotated either when the size crosses 100M or daily. For this to complete, the logrotate command should run successfully. 
By default, the entry inside /etc/cron.daily/logrotate should run the daily logrotate command which in-turn should rotate the logs. Please update the value in `maxsize` depending upon the disk space availability.

If for some reason, we need to manually rotate /var/log/messages, run the below command.

     sudo /usr/sbin/logrotate /etc/logrotate.d/messages 

 

2. Mounting /var/log/messages: mount /var/log/messages to a separate volume with enough disk space.