Error: "No VMs" is displayed and storage policies are not visible on the Configure Resources page during vApp creation

Products

• VMware Cloud Director

Issue/Introduction

• The vApp creation wizard displays "No VMs" and and "There are no disks" at the VM selection stage.
• After a Tenant selects NEW > 'Add vApp from Catalog' the modal window appears but on the Configure Resources page it's blank and nothing is visible to select.
• After navigating to Content Hub > Content > vApp Templates and selecting CREATE VAPP the modal window appears but on the Configure Resources page it's blank and nothing is visible to select.
• The wizard spins indefinitely and does not allow the user to proceed.
• Unable to deploy vApps due to storage profiles not loading in UI.
• In the Tenant UI navigating to Data Centers > Storage > Storage Policies shows a spinning wheel which spins for many minutes before stopping and showing no policies.
• The issue is intermittent; a vApp template may show VMs correctly at one moment and fail shortly after.
• Adding VMs individually to an existing vApp works correctly.
• Browser Developer Tools may show long-running or timed-out API calls related to storage profiles or datastore details.

Environment

• VMware Cloud Director 10.6.1.x

Cause

This issue is caused by performance degradation in the Cloud Director Database. Views are queried when fetching storage profile and datastore overlap details during the vApp deployment wizard. If the database execution takes too long, the UI times out, resulting in an empty VM list or a hanging wizard.

Resolution

To ensure optimal performance, it is recommended to reclaim disk space and update database statistics.

Online Optimization

If a maintenance window is not immediately possible then updating the statistical data on the following database tables may provide temporary mitigation:

1. Perform a backup as outlined in Backup and Restore of Your VMware Cloud Director Appliance.
2. Connect with SSH to the Cloud Director database primary cell
3. Connect to the database on the primary cell:
   
   sudo -i -u postgres psql vcloud
    
   (or)
   
   db
   
   
4. Run the ANALYZE command to update statistical data about the contents of database tables. The query planner then uses these statistics to determine the most efficient execution plan for running queries.
   
   ANALYZE datastore_profile_inv;
ANALYZE storage_profile_inv;
ANALYZE datastore_inv;
   
   
5. Return to the Cloud Director UI and refresh the Web Browser page.
6. Review to confirm if the issue is resolved.

Offline Optimization

This workflow requires downtime for the Cloud Director vmware-vcd service on all cells at the same time. This result in the Cloud Director UI becoming unavailable.

Note: That VACUUM FULL requires a table-level lock and should be performed during a maintenance window.

1. Perform a backup as outlined in Backup and Restore of Your VMware Cloud Director Appliance.
2. Connect with SSH to ALL of the Cloud Director cells in the environment.
3. Stop the services on all cells:
   
   /opt/vmware/vcloud-director/bin/cell-management-tool cell -i $(service vmware-vcd pid cell) -s
   
   
4. Connect to the database on the primary cell:
   
   
   sudo -i -u postgres psql vcloud
    
   (or)
   
   db
   
   Note: To determine which cell is the primary cell, the following command could be run on any of the database cells SSH console.
   
   repmgr cluster show


5. Perform a vacuum of the database to reclaim space and analyze the contents of the tables. Note: These command may take a few minutes to run in larger database.
    
   vacuum full;
vacuum analyze;


6. Exit the database connection by using CNTRL+D key press.
7. Start the vmware-vcd service on the first cell.
   
   systemctl start vmware-vcd
   
   
8. Repeat step 7 on the remaining cells once the first cell has come online.
   
   Note: To monitor the service startup the following command can be used:
   
   tail -f /opt/vmware/vcloud-director/logs/cell.log