Cloud Director Standby or Application Cells cannot connect to the database on the Primary Cell

Products

• VMware Cloud Director

Issue/Introduction

• Cloud Director (VCD) portal / UI / API is inaccessible.
• A vSphere outage occurred which affected the VCD Cells VMs such as a storage issue.
• The vmware-vcd service can only start successfully on some Cells.
• VCD Cells are not all on the same network / subnet and require static routes to connect to each other's eth1 interface.
• Standby or Application Cells cannot connect to or ping the Primary Cell.
• vApp Options on the Cell VMs in vSphere are populated, including required routing set for the vcloudnet.routes0.VMware_vCloud_Director and/or the vcloudnet.routes1.VMware_vCloud_Director keys.
• Running the ovfenv command on the Cells does not show these expected routes:

ovfenv --key vcloudnet.routes0.VMware_vCloud_Director
ovfenv --key vcloudnet.routes1.VMware_vCloud_Director

• Running the ip route command on the Cells does not show the expected static routes for eth0 and/or eth1:

ip route

• The /opt/vmware/etc/vami/ovfEnv.xml file is missing or empty on the VCD Cells:

ls -l /opt/vmware/etc/vami/ovfEnv.xml

Environment

VMware Cloud Director

Cause

If there is an issue with vSphere during the initial Cloud Director Cell VM power on, then the vApp Options may not be passed into the guest OS and the ovfenv populated.
The VCD Cells will attempt to set any static routes based on the ovfenv values for vcloudnet.routes0.VMware_vCloud_Director and/or the vcloudnet.routes1.VMware_vCloud_Director keys.

If the Cells are missing required routes then network connection between the Cells on different subnets may fail.

Resolution

To resolve the issue stop and start the Cloud Director Cell VMs in turn from vSphere to ensure they set the vApp Options and populate the ovfenv on startup.
NOTE: A guest OS reboot is not sufficient. The entire VM needs to enter a powered off status and then be powered on again in order for /opt/vmware/etc/vami/ovfEnv.xml to be populated within the guest. 

Confirm that the ovfenv for routes is populated for eth0 and eth1 respectively by running the following commands:
NOTE: these commands will return no result if no optional static routes were set in the vApp Options in vSphere.

ovfenv --key vcloudnet.routes0.VMware_vCloud_Director
ovfenv --key vcloudnet.routes1.VMware_vCloud_Director

To confirm the ovfenv values have been applied to the Cell's network configuration run the following command:

ip route

Additional Information

For more information on configuring static routes for Cloud Director Cells see the documentation, Edit the Static Routes for Your VMware Cloud Director Appliance Network Interfaces.

For more information on vApp Options in vSphere see the documentation, Virtual Machine vApp Options

For issues with static routes encountered after upgrading Cloud Director see the article, Cell connectivity is lost after upgrading VMware Cloud Director.