Cloud Director service becomes inaccessible after a period of time

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Cannot access Cloud Director portal after a period of time
• Cloud Director service becomes unstable or stays down
• You can see similar these errors in jmx.log, jms,log, or postgresql.log:
  • Cannot allocate memory
  • Too many open files
• You can see logs about periodical accessing from a load balancer in console-proxy.log.

Environment

VMware Cloud Director 10.x
VMware Cloud Director for Service Provider 10.x

Cause

This issue is caused by accumulation of sockets from a load balancer connecting to TCP Port 8443, the port used by console proxy service.
As a result, this accumulation leads to an exhaustion of file descriptors and memory.

Resolution

This is a known issue in Cloud Director 10.1 and later.

Workaround:

1. Load balancers should monitor only TCP Port 443, the port used by the HTTP and API Interface of the Cell.
2. The API endpoint for monitoring whether this service is up should also be leveraged.
   1. HTTP Request: GET /api/server_status
   2. HTTP Response: Service is up.

Additional Information

For information on how to Load Balance Cloud Director with NSX-T, you can refer to the link below:
Load Balancing vCloud Director with NSX-T