No space left on device /tmp/sddc-201912301743/var/log/vmware/vcf/lcm/lcm.log

Products

• VMware Cloud Foundation

Issue/Introduction

• The purpose of this KB article would be to provide instructions on how to delete older extracted log files in such a scenario.

Symptoms:

• LCM ensures that day older logs are stored in compressed format and are restricted to 30 days or 2GB in total. However, these compressed log files can be extracted during debugging processes and left in the same log folder.
• These older log files, if left without cleaning up, then occupy too much space and cause issues for SOS log collection. Example of SOS log collection failure below:

----------------------------------------------------------------------------------
2019-12-30T17:46:09.537Z [ERROR commandutils.py::execute_cmd_locally::230::collect_sddc_manager_logsThread2] rsync: write failed on "/tmp/sddc-201912301743/var/log/vmware/vcf/lcm/lcm.log": No space left on device (28)
----------------------------------------------------------------------------------

Environment

VMware Cloud Foundation 3.9.x

Resolution

• Command to view such files : find /var/log/vmware/vcf/lcm -iname "*.log" -atime +1 -type f
• Command to delete such files : find /var/log/vmware/vcf/lcm -iname "*.log" -atime +1 -type f -delete

NOTE:- These commands show and delete only the logs files that have been accessed (-atime flag) more than a day (+1 flag) ago. This ensures it does not delete the log files that are currently being written. It also ensures the compressed log files stored (for 1 day older logs) are not deleted