Error: “503 Service Unavailable” in VMware Cloud Director during API or Console Operations

Products

• VMware Cloud Director

Issue/Introduction

• Intermittent HTTP 503 (Service Unavailable) errors are observed in the VMware Cloud Director UI and API.
• Cloud Director becomes temporarily unresponsive during periods of monitoring, backup, or API activity.
• One or more Cloud Director cells reach saturation causing the 503 errors and causing VCD to become unresponsive, while other cells remain underutilized.
• The issue is most visible during automated polling or backup windows.
• VMware Cloud Director (VCD) performance problems are observed, including slow UI response, failed API calls, and temporary service unavailability.
• Log excerpts observed during impact:
  
  ERROR: duplicate key value violates unique constraint "uq_orgmemberid_identifiertoken"
org.springframework.orm.ObjectOptimisticLockingFailureException
org.hibernate.StaleStateException

Environment

VMware Cloud Director 10.6

Cause

This issue can occur when incoming API or monitoring traffic is persistently routed to a single Cloud Director cell by a load balancer.
Client IP–based persistence causes repeated API requests from monitoring or backup tools to be sent to the same cell.
The affected cell exhausts its database connection pool, resulting in HTTP 503 responses.
Other Cloud Director cells remain available but are not utilized due to load-balancer persistence behavior.

Resolution

1. Identify cells returning HTTP 503 responses
   
   On each Cloud Director cell, run:
   
   grep 'HTTP/1\.[01]" 503' /opt/vmware/vcloud-director/logs/*request.log | wc -l

Check for cells reporting significantly higher counts.
   
   
2. Verify load balancer configuration
   

   From the load balancer configuration in front of Cloud Director:

   • Review the Persistence Profile applied to the VIP or pool.

     • NSX > Load Balancing > Virtual Services  and click the '>' icon to expand the section.

       Deactivate 'Persistence'

3. Validate Resolution

   • After adjusting load balancer settings:

     • Re-run the HTTP 503 checks from Step 1 on all cells.

     • Confirm that database connections remain below per-cell limits.

     • Verify Cloud Director UI and API responsiveness during monitoring or backup activity.

Additional Information

VMware NSX-T Data Center - Add a Persistence Profile

VMware Avi Load Balancer - Persistence