Error: "Error caused vCenter Server disconnect" occurs and vCenter Server shows as disconnected

Products

VMware Cloud Director

Issue/Introduction

Cloud Director(VCD) shows vCenter Server in a disconnected state in the Provider UI under Resources > Infrastructure Resources > vCenter Server Instances.
Cloud Director reports errors of the form:

Error caused vCenter Server disconnect
- org.apache.http.conn.HttpHostConnectException: Connect to vcenter.example.com:443 [vcenter.fqdn/vcenter.ipddr] failed: Connection timed out (Connection timed out)
- Connect to vcenter.example.com:443 [vcenter.fqdn/vcenter.ipddr] failed: Connection timed out (Connection timed out)
- Connection timed out (Connection timed out)

Attempting to reconnect or edit vCenter Server through the Cloud Director UI report errors such as:

Error caused vCenter Server disconnect
Failed to connect

Restarting the Cloud Director services or the Cloud Director cells has no effect on the issue.
Making a curl request from the Cloud Director cells to the vCenter Server fails or times out:

curl -k -v https://vcenter.example.com:443
* Trying vcenter.example.com:443...
* connect to vcenter.example.com port 443 failed: Connection timed out
* Failed to connect to vcenter.example.com port 443 after 130071 ms: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to vcenter.example.com port 443 after 130071 ms: Connection timed out

Making a curl request from the vCenter Server to the Cloud Director Cells fails or times out:

curl -k -v https://vcd-cell.example.com
* Trying vcd-cell.example.com:443...
* connect to vcd-cell.example.com port 443 failed: No route to host
* Failed to connect to vcd-cell.example.com port 443 after 3073 ms: No route to host
* Closing connection 0
curl: (7) Failed to connect to vcd-cell.example.com port 443 after 3073 ms: No route to host
Performing a vCenter refresh triggers the following error in VCD cell log file /opt/vmware/vcloud-director/logs/vcloud-container-debug.log

ERROR | Backend-activity-pool-7910 | VirtualCenterService | Error occurred while connecting to VC

Environment

VMware Cloud Director 10.x

Cause

This issue can occur if there is a network communication issue that stops traffic from vCenter Server reaching the Cloud Director cells.

Resolution

To resolve the issue ensure network traffic from vCenter Server can reach the Cloud Director cell(s).

Firstly review the networking routes on the vCenter Server and confirm their validity. To view the routes on the vCenter Server SSH to it as root, enable the shell and run the command:

ip route

If any configuration issues are found then refer to Configuring Persistent Static Routes on VCSA for information on how to add routes. Once the routing on the vCenter Server is set correctly we can test that the communication is successful.
Confirm that the vCenter Server can reach the Cloud Director cells using curl:
1. SSH to the vCenter Server, log in as root, and enable the Shell.
2. Use curl to connect to the Cloud Director cell and confirm this is successful, for example:curl -v telnet://vcloud.example.com:443
  * Host vcloud.example.com::443 was resolved.
  * IPv6: (none)
  * IPv4: 192.###.##.##
  * Trying 192.###.##.##:443...
  * Connected to vcloud.example.com: (192.###.##.##) port 443
Confirm that the Cloud Director cells can reach vCenter Server using curl:
1. SSH to the Cloud Director Cell and log in as root.
2. Use curl to connect to the vCenter Server and confirm this is successful, for example:
  
  curl -k https://vcenter.example.com/
  
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
  <html xmlns="http://www.w3.org/1999/xhtml" lang="en">
  ...
If network communication issues are still occurring between the vCenter Server and Cloud Director cells then engage with your local networking team for their review.

Additional Information

Impact/Risks: Cloud Director will encounter issues such as task execution failures if it cannot connect to vCenter Server.

VMware Ports and Protocols