Cloud Director Cell Partition Full

Products

VMware Cloud Director

Issue/Introduction

You are unable to access the VMware Cloud Director (VCD) provider or tenant UI. When checking the VCD cell appliance via SSH, you observe one or more of the following symptoms:

Running the df -h command shows that the root (/) or /opt partition is at or near 100% full.
The vmware-vcd service fails to start, respond, or crashes.
Log files stop writing new messages.
Cells may appear as "Inactive" in the Provider Portal.
Log bundle creation fails.

Environment

VMware Cloud Director 10.4 and later

Cause

This issue is typically caused by an excessive accumulation of log files, diagnostic bundles, or system dumps that are not being properly rotated or cleaned up.

Resolution

Starting Point

The first step is to identify which files are consuming the space. A useful command to find the largest files is:

du -aShx / | sort -rh | head -20

Based on the files consuming the disk space, see the appropriate article below for the specific resolution.

Known Issues Causing A Full root Partition

Log Retention Policy

Symptom: The /opt/vmware/vcloud-director/logs/ directory is full of many request.log files (e.g., yyyy_mm_dd.request.log).
Resolution: Cloud Director cell disk space is full due to the accumulation of request.log files (KB 325687)

`.hprof` Java Dump Files

Symptom: You find large java_pid****.hprof files in the /opt/vmware/vcloud-director/logs/ directory, which are Java memory dumps.
Resolution: Cloud Director root partition full due to hprof files (KB 330107)

Improper `rsyslog` rotation

Symptom: The /var/log/messages file is not rotating correctly, especially in VCD 10.6 and later where auth.log is merged.
Resolution: Log files consuming excess amount of storage space in VMware Cloud Director 10.5 and 10.6 (KB 328001)

Single `cell.log` File Growth

Symptom: The main cell.log file has grown to an excessively large size and is not being rotated.
Solution: VMware Cloud Director root partition full due to cell.log (KB 374009)

Improper `logrotate` config

Symptom: The /opt/vmware/var/log/vcd/vcd_ova_ui_uwsgi.log or /var/log/nginx/access.log files are excessively large due to frequent API calls and improper log rotation settings.
Resolution: VMware Cloud Director root ("/") partition becomes full due to large /opt/vmware/var/log/vcd/vcd_ova_ui_uwsgi.log and /var/log/nginx/access.log. (KB 411155)

Fallback Solution: Expand Partition

If cleaning files is not enough and the partitions are undersized, you must expand the virtual disk.

Recommended Article: Extending root disk of Cloud Director appliance (KB 380554)

Additional Information

Log Bundle Generation

Risk: The diagnostic script (vmware-vcd-support -m -l) creates the log bundle in the current working directory you are in when you execute it. If you run this command from the root partition (e.g., / or /root), you risk filling up the root partition completely, which can crash the cell.

If the root partition on the cell is at risk of filling up if several gibabytes of data were to be added to the partition, it is recommended to navigate to the transfer partition before running the script. This partition is typically sized with more space. Verify it has sufficient free space (e.g., 10GB or more) to avoid impacting cell-to-cell transfers or template uploads.

cd /opt/vmware/vcloud-director/data/transfer

The df -h command will show the free space.

Reference: Collecting diagnostic information for VMware Cloud Director (KB 320409)