/log/retained-logs/ partition is 100% full

Products

• VMware Telco Cloud Automation

Issue/Introduction

• There were many kubelet log files generated in /logs/retained-logs/kubelet.service and /log partition of the TCA-CP became 100% full. 

Environment

3.1.1

3.2

Cause

• The size of kubelet log files were larger than usual. These logs typically will not grow beyond a few MB in a day. 
• Currently the log rotation is defaulted to once per day. There is no capability to rotate it based on size.

Resolution

1. Restart the kubelet service
   systemctl restart kubelet
   
   
2. Restart the containerd service
   systemctl restart containerd

Additional Information

1. It is important to investigate whether any issues are causing the larger log files.
   Manual log file deletion is definitely recommended.
   
   
2. You can find the large log files using the command below, then delete (or move) them.
   find /logs/retained-logs -type f -size +250M
   
   
3. You can change the above command to more than 100M as well.