Cloud Director cell disk space is full due to the accumulation of request.log files.

Products

• VMware Cloud Director

Issue/Introduction

• VMware Cloud Director logs the incoming API requests to the VMware Cloud Director cells in request.log files. 
• Logs with the format "yyyy_mm_dd.request.log" which are located in the directory /opt/vmware/vcloud-director/logs/ are filling up the systems disk space.
• The request logs are not being removed after setting a retention policy with the below command:

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vcloud.http.log.retainDays -v 10

• Log bundle creation fails.
• Root partition shows 100%.
• Unable to access Cloud Director user interface.
• VCD cells shows "Inactive" in the provider portal.
• Within the log file /opt/vmware/vcloud-director/logs/jms.log you observe a warning message similar to the following:

yyyy-mm-dd xx:xx:xxxxx | WARN     | is.core.server.impl.ActiveMQServerImpl$6@2a5fb60a) | server  | AMQ222210: Free storage space is at 0B of x.xGB total. Usage rate is 100.0% which is beyond the configured <max-disk-usage>. System will start blocking producers.

Environment

VMware Cloud Director 10.4

VMware Cloud Director 10.5

Cause

This issue occurs as the log retention parameter vcloud.http.log.retainDays is not being correctly read from the Cloud Director database.

Resolution

This issue is resolved in VMware Cloud Director 10.6, available at Broadcom Downloads.

If you are unable to upgrade, please see the workaround below.

Workaround:

Follow the below steps to manually update global.properties file to retain the request.log for a specific period of days on the cloud director cells.

 

1. Backup the file /opt/vmware/vcloud-director/etc/global.properties on the Cloud Director cell.

2. Insert the below line in the file /opt/vmware/vcloud-director/etc/global.properties and specify the amount of days you wish to retain the logs. The below example sets the retention to 10 days.

vcloud.http.log.retainDays=10

3. Restart the vmware-vcd service on the Cloud Director cell to apply the change. For more information see Managing a Cell.

              

        systemctl restart vmware-vcd.service

 

4. Repeat steps 1 - 3 above on all Cloud Director cells in the environment. 

Note: The cleanup runs everyday at midnight as per the time on the system where the cell is running.

Additional Information

If immediate cleanup is required, space can be reclaimed by removing the old request.log files located in the /opt/vmware/vcloud-director/logs directory using the below command:

rm YYYY_MM_DD.request.log

Example: rm 2024_08_25.request.log

In cases where the Cloud Director portal becomes entirely inaccessible due to severe cell disk space issues, more advanced cleanup steps are detailed in:Cloud Director Portal becomes inaccessible due to cell space issues