VMware Cloud Director GUI Inaccessible and vApps Status "Unknown" After Database Maintenance

Products

• VMware Cloud Director

Issue/Introduction

• After executing a database inventory cleanup script in VMware Cloud Director (VCD), the environment exhibits the following symptoms:
• Administrators and tenants are unable to log in to the VCD Provider or Tenant portals.
• Existing vApps are displayed with an "Unknown" status in the inventory.
• The VCD environment is unstable; vmware-vcd services on cell members may fail to start or crash repeatedly.

Environment

VMware vCloud Director

Cause

This issue occurs when database maintenance scripts are executed while VCD cells in the server group are still active (online).

Cleanup scripts are designed to purge historical data, such as expired tasks, audit logs, and events directly from the database. If VCD cells are running during this process, they maintain active database connections and hold in-memory references to the objects the script is attempting to modify or delete.

This conflict creates a race condition, resulting in:

• Database concurrency violations.
• Data inconsistency between the cell's in-memory state and the database records.
• Orphaned records that cause the "Unknown" status for vApps.

Resolution

To prevent this issue, the VCD service must be stopped on all cells in the server group before executing any database maintenance scripts.

Correct Procedure for Inventory Cleanup

• Schedule a maintenance window, as this procedure will cause a complete outage of the VCD service.
• Stop VCD Services: Log in to each VCD cell and gracefully stop the vmware-vcd service.
  Command: /opt/vmware/vcloud-director/bin/cell-management-tool -u <admin_username> cell --shutdown
• Alternatively, use service vmware-vcd stop or systemctl stop vmware-vcd 
• Verify Shutdown: Ensure the service is stopped on all cells in the server group.
• Rerun cleanup script
• Restart Services: Once the script has completed successfully, restart the vmware-vcd service on all cells, one at a time.

Important: If you have already run the script while cells were active, the database may be in a corrupted state. The recommended fix is to follow the steps above (stop all cells) and re-run the cleanup script. This allows the script to complete its operation on a static database, which may resolve the inconsistencies. If the issue persists, you may need to restore the VCD database from a healthy backup taken before the script was run.