Error: "Http failure response for <API Call>: 503 Service Unavailable" when managing Virtual Machines in the VMware Cloud Director UI

Products

• VMware Cloud Director

Issue/Introduction

• When attempting to manage, create, or delete Virtual Machines (VMs) or vApps, the UI may fail to load with the following error in Cloud Director:

HTTP failure response for <API Call>: 503 Service Unavailable

• VMware Cloud Director UI randomly generates errors as below while interacting with the VMs or the vApps:
  
  "0 Unknown error" from the following request "request": {
    "method": "GET",
    "url": "https://VCD-URL/api/query?type=vm&format=records&type=vm&page=1&pageSize=15&filterEncoded=true&filter=(((isExpired==false));((isVAppTemplate==false;vdc==xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx)))&sortDesc=dateCreated&fields=metadata%40SYSTEM%3Avmware.cloud.director.availability.protected%2Cmetadata%40GENERAL%3Avmware.cloud.director.ui.badges.blue%2Cmetadata%40GENERAL%3Avmware.cloud.director.ui.badges.red%2Cmetadata%40GENERAL%3Avmware.cloud.director.ui.badges.orange%2Cmetadata%40GENERAL%3Avmware.cloud.director.ui.badges.green%2Cmetadata%40GENERAL%3Avmware.cloud.director.ui.badges.magenta&links=true"
}

Environment

VMware Cloud Director 10.6.x 

Cause

• This issue arises when the database connection pool is overloaded, resulting in a lack of available connections to process incoming requests.
• This can be confirmed by checking the following logs: /opt/vmware/vcloud-director/logs/vcloud-container-debug.log and /opt/vmware/vcloud-director/logs/vcloud-container-debug.log/cell-runtime.log.
• The underlying cause is indicated by the error:

org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-jetty-17471] Timeout: Pool empty. Unable to fetch a connection in 20 seconds, none available [size:75; busy:75; idle:0; lastwait:20000].

Resolution

To resolve this error:

1. Verify that the allocated vCPUs and virtual memory for all the VCD cells align with the VMware Cloud Director Appliance Sizing Guidelines.
2. Ensure that an adequate number of cells (both Standby and Application cells) are deployed. For more information, refer to the : VMware Cloud Director Appliance Deployments and Database High Availability Configuration documentation.