VMware Cloud Director shows three Primary Nodes

Products

• VMware Cloud Director

Issue/Introduction

• A failover occurred and the VMware Cloud Director (VCD) cluster health is currently reported as DEGRADED or READ ONLY in the VAMI. To access the VAMI, log in as root to the appliance management UI at https://<primary_eth1_ip_address>:5480.
• On the VAMI you see three primary cells: two with status failed and one with status running. 
• An automatic failover was triggered by performing an action such as rebooting the primary cell in a VCD appliance cluster
• VCD cells appear to be in a "split-brain" scenario with three primary cells listed.
• The Cloud Director request logs on the running primary cell shows an high number of api calls coming from a specific IP Address similar to:

  <IP_ADDRESS> - - [<DATE>:<TIME> +0000] "GET /api/query?type=adminVM&filterEncoded=true&filter=(container==urn:vcloud:vapp:########-####-####-####-############)&format=records HTTP/1.1" 200 3511 "-" "-" 15

Environment

VMware Cloud Director 10.6.x

Cause

The high number of api calls on the primary will use all the connection pool, causing the primary node to be unresponsive and generating an automatic switchover to a standby cell. This will be repeated also for the new primary, which in turn will go in failed status and switchover to  remaining standby node, leaving 3 nodes all primary, but two of them in failed status. Those high number of api calls are generated from a specific IP address which may be associated to 3rd party applications (i.e. backup application).

Resolution

To help resolving the issue, you will need to do the following:

1. Check the /opt/vmware/vcloud-director/logs/request.log file and verify which IP address is generating excessive number of api calls.
2. Change the HA Cluster failover mode from AUTOMATIC to MANUAL to avoid further switchover due to excessive api calls. See article with steps: INDETERMINATE Cluster failover status
3. Unregister a Failed VMware Cloud Director Appliance Primary or Standby Cell in a Database High Availability Cluster
4. If the IP Address is from a specifc 3rd party application, open a ticket with the vendor to help addressing the issue with those api call or make sure that the api calls are Load Balanced to avoid the cell to become unresponsive.
5. Once the issue with excessive api calls has been addressed to change back the HA Cluster failover to AUTOMATIC.