Cloud Director Portal becomes inaccessible due to cell space issues

Products

• VMware Cloud Director

Issue/Introduction

• Cloud Director Portal becomes inaccessible after a period of time.
• Running df -h on a Cell shows high usage of disk space
• The /opt/vmware/vcloud-director/logs/cell.log shows IO Exceptions for the current service start attempt.

 

Environment

Cloud Director 10.x

Cause

If a cell has been operating for a period of time and suddenly become unresponsive, it highly likely that it has encountered space issues.

Certain services will need to be able to write to disk, and an inability to do so will cause the VCD service to crash.

Resolution

To verify if you are encountering space issues, 

1. SSH to the Cell.
2. Run df -h
3. Space issues are generally on the / partition.

 

Space on a Cloud Director Cell can be consumed for a number of reason.

To verify what files are taking up the most space, run the following:

find <directory> -type f -size <size of files> | grep -v "/opt/vmware/vcloud-director/data/transfer"

Example:

• find / -type f -size +100M | grep -v "/opt/vmware/vcloud-director/data/transfer"
• Starting at the root directory, find any files larger than 100M

 

1. A Known Issue with Request logs not being correctly purged
   1. https://knowledge.broadcom.com/external/article/325687/cloud-director-cell-disk-space-is-full-d.html
2. Journal Logs can also take up a lot of space
   1. journalctl --vacuum-time=2d
   2. Remove any journal logs older than 2 days
3. Login logs can also take up a lot of space
   1. cat /dev/null > /var/log/wtmp
   2. This will clear any references to logins
4. Java Memory issues which generate HPROF files
   1. This is generally less encountered as Appliance Sizing Guidelines now exist.
   2. These will be located in /opt/vmware/vcloud-director/logs directory.
   3. Remove these using rm if you are sure they are not of use.

Once you have cleared the files taking up space, start or restart the Cloud Director Service depending on the current state of the service.

• systemctl start vmware-vcd
• systemctl restart vmware-vcd