vCenter Server Root partition (/dev/sda3) available space is consumed by lighttpd logs

Products

• VMware vCenter Server

Issue/Introduction

This article provides step to mitigate the effects of lighttpd logs consuming all available space in /dev/sda3.

Symptoms:

Root partition is 80% to 100% full as reported by VCSA VAMI or command line (df). vCenter may not be able to connect due to services not starting.
 

This has been reported to occur after an upgrade to 6.7.
 

Investigating the filesystem’s used space:

# df -h
Filesystem  Size  Used  Avail Use%  Mounted on
[…]
/dev/sda3   11G   11G   0     100%  /
[…]

List large files (over 20MB):

# find / -type f -size +200000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'
/opt/vmware/var/log/lighttpd/access.log-<date>: 1.8G
/opt/vmware/var/log/lighttpd/access.log: 2.9G

 

Environment

VMware vCenter Server 6.7.x

Resolution

Workaround:

1. Navigate to the lightpd service log directory.

# cd /opt/vmware/var/log/lighttpd
 

2. Investigate and take note of the number of files and their sizes:

#ls -lart
 

3. Determine which files are consuming excessive space. Make copies if needed.

4. Delete the contents of the offending files without deleting the files:

#cat /dev/null > <filename>
 

For example:

#cat /dev/null > access.log
 

5. Restore vCenter functionality by restarting vCenter services

# service-control –start
 

6. If the services do not start and the vCenter has been rebooted since the root partition was full, the other necessary partitions may not be mounted. Rebooting the VCSA will resolve this issue.

Additional Information

Impact/Risks:

The following resolution will delete the log messages from the lighttpd service. The lighttpd service runs the VMware Appliance Management Interface (VAMI).

If additional analysis is required, make copies of the access.log files contained in /opt/vmware/var/log/lighttpd to another partition or another system.