VMware Cloud Director cell disk space is full due to the accumulation of API request log files on the root ("/") partition
search cancel

VMware Cloud Director cell disk space is full due to the accumulation of API request log files on the root ("/") partition

book

Article ID: 325687

calendar_today

Updated On:

Products

VMware Cloud Director

Issue/Introduction

  • VMware Cloud Director(VCD) logs the incoming API requests to the VCD cells in request.log files. 
  • Logs with the format "yyyy_mm_dd.request.log" which are located in the directory /opt/vmware/vcloud-director/logs/ are filling up the systems disk space.
  • The API request logs are not being removed after setting a retention policy with the below cell management tool (CMT )command:

    /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n "vcloud.http.log.retainDays" -v <new value>
  • Log bundle creation fails.
  • Root partition shows 100%.
  • Unable to access Cloud Director user interface.
  • VCD cells shows "Inactive" in the provider portal.
  • The vmware-vcd service fails to start with the message 

    Failed to start LSB: The VMware vCloud Director service.

  • Within the log file /opt/vmware/vcloud-director/logs/cell.log it may show:

    Found 1 deadlock 

  • Within the log file /opt/vmware/vcloud-director/logs/jms.log you observe a warning message similar to the following:

    yyyy-mm-dd 14:15:51| WARN | is.core.server.impl.ActiveMQServerImp1$6@2a5fb60a) | server | AMQ222210: Free storage space is at 0B of 15.0GB total. Usage rate is 100.0% which is beyond the configured <max-disk-usage>. System will start blocking producers.
  • Unable to upload and deploy templates due to error 

    Error - Cannot retry request with a non-repeatable request entity - Broken pipe (Write failed)

Environment

VMware Cloud Director 10.4.x
VMware Cloud Director 10.5.x
VMware Cloud Director 10.6.x

Cause

This issue occurs when a retention policy for API request logs is either not configured or if the retention policy is configured too high. By default a retention policy is not applied. This allows the API request log files to accumulate indefinitely until all available disk space is consumed on the partition.

Resolution

To resolve this issue then configure a retention policy on the API request logs by following the process to Control the API Request Logging in VMware Cloud Director. 

Note: The automatic cleanup runs everyday at midnight as per the time on the system where the cell is running. 

If immediate cleanup is required, space can be reclaimed by manually removing old API request logs from the system. To do this:

  1. Open an SSH session to the impacted Cloud Director cell and login as the root user.
  2. Run this command to confirm the current usage on the root ("/") partition.

    df -h

  3. Navigate to the log directory /opt/vmware/vcloud-director/logs/ using the below command:

    cd /opt/vmware/vcloud-director/logs/

  4. Review the the size and amount of API request logs present on the system using this command:

    ls -lh *request.log

  5. Assess which API request files can be removed and then proceed to do. It would be recommended to begin with removal of the oldest API request logs and retain the most recent API request log. 
    Here is an example of using the rm command to remove an API request log file name 2025_08_25.request.log.

    rm 2025_08_25.request.log

Note: In Cloud Director 10.5.x an issue existed whereby the vcloud.http.log.retainDays setting which was applied using the cell management tool did not work. This issue was resolved in Cloud Director 10.6.0 which is available at VMware Cloud Director product downloads. 

To workaround that issue where the setting applied using the cell management tool was not rotating API logs, the following can be performed. 

  1. Open an SSH session to the impacted Cloud Director cell and login as the root user.
  2. Backup the file /opt/vmware/vcloud-director/etc/global.properties on the Cloud Director cell.

    cp /opt/vmware/vcloud-director/etc/global.properties /opt/vmware/vcloud-director/etc/global.properties.backup

  3. Open the file for editing with vi.

    vi /opt/vmware/vcloud-director/etc/global.properties

  4. Insert the below line in the file and specify the amount of days you wish to retain the logs. The below example sets the retention to 10 days.

    vcloud.http.log.retainDays=10

  5. Restart the vmware-vcd service on the Cloud Director cell to apply the change. For more information see Managing a Cell.

    systemctl restart vmware-vcd.service

  6. Repeat steps 1 - 5 on all Cloud Director cells in the environment.

Additional Information

More advanced space cleanup steps are detailed in: Cloud Director Portal becomes inaccessible due to cell space issues

Powered by Wolken Software