LCM service fails to start due to /dev/mapper/data_vg-vmware filesystem being full

Products

• VMware SDDC Manager

Issue/Introduction

The LCM service fails to start due to /var/log/vmware partition being full.

Checking the space utilization using "df -h" shows /var/log/vmware partition usage at 100%

root@test-vcf-kb [ ~ ]# df -h
Filesystem                          Size  Used Avail Use% Mounted on
devtmpfs                            7.9G     0  7.9G   0% /dev
tmpfs                               7.9G   20K  7.9G   1% /dev/shm
tmpfs                               7.9G  1.1M  7.9G   1% /run
tmpfs                               7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda4                            28G   25G  1.6G  95% /
tmpfs                               7.9G  9.4M  7.9G   1% /tmp
/dev/sda2                           120M   27M   85M  24% /boot
/dev/mapper/data_vg-vcf             181G  1.7G  170G   1% /opt/vmware/vcf
/dev/mapper/data_vg-vmware           63G  2.3G   90M 100% /var/log/vmware
/dev/mapper/data_vg-data            7.9G  160M  7.3G   3% /data
/dev/mapper/lcmmount_vg-nfs--mount  503G  165G  313G  35% /nfs/vmware/vcf/nfs-mount
tmpfs                               1.6G     0  1.6G   0% /run/user/1000

 

Environment

VMware Cloud Foundation 5.x

Cause

The /var/log/vmware/SDDC-Manager directory was utilizing about 58G leading to space being full.

Resolution

To clean up unwanted or large files:

1. Open an SSH session to the SDDC Manager appliance.
2. Login as vcf and switch to root using "su -"
3. Navigate to the VMware log directory:
   

   cd /var/log/vmware/

4. Check for large files and directories:

   du -h / --max-depth 1

5. Identify any old SOS or backup files that can be safely removed to free up space by running the below command:
   

   rm -rf <enter directory or file name>
   Example:
   rm -rf sddc-xxx.tgz
   rm -rf sos-xxx.7z
   rm -rf file.log