"Timeout has occurred" error when configuring On-Premises to Cloud Replciations in Cloud Director Availability

Products

VMware Cloud Director

Issue/Introduction

Symptoms:

Configuring On-Premises to Cloud replications in Cloud Director Availability fail with the error:

Timeout has occurred

In the /opt/vmware/h4/cloud/log/cloud.log file on the Cloud Replication Management Appliance, you see entries similar to:

2020-09-09 07:49:28.575 ERROR - [UI__2ad66003-ac7f-4028-861e-d7cd22577da1_3c] [https-jsse-nio-8443-exec-10] c.v.h.c.c.error.ExceptionAdvisorBase     : A GET request from root[ 192.168.1.50] to /vc-sites/site-name/vcenters failed.

com.vmware.exception.ReadTimeoutException: Read timed out
    at com.vmware.exception.converter.ClientExceptionConverter.convertException(ClientExceptionConverter.java:40)
    at com.vmware.rest.client.AbstractRestClient.genericExchange(AbstractRestClient.java:149)
    at com.vmware.rest.client.json.RestClient.exchange(RestClient.java:97)
    at com.vmware.rest.client.json.RestClient.exchange(RestClient.java:128)
    at com.vmware.rest.client.json.RestClient.exchange(RestClient.java:171)
    at com.vmware.h4.manager.client.ManagerClient.browseVcenters(ManagerClient.java:1466)
    at com.vmware.h4.cloud.peer.VcenterInventoryBrowser.browseVcenters(VcenterInventoryBrowser.java:47)
    at com.vmware.h4.cloud.controller.VcenterSiteController.browseVcenters(VcenterSiteController.java:160)

In the /opt/vmware/h4/replicator/log/replicator.log file on the Cloud Director Availability On-Premises Appliance, you see entries similar to:

2020-09-09 07:05:41.635 WARN - [UI__2e6a8e29-2206-42be-a5c1-a849734f09e1_Es_UX-lZ] [https-jsse-nio-8043-exec-3] c.v.h.r.vc.inventory.VcInventoryService : Couldn't talk to VC ServiceEndpoint{serviceId='b335f0e2-7657-40eb-85cb-42be2eb28a26', instanceId='b335f0e2-7657-40eb-85cb-42be2eb28a26', uri=https://vcsa.example.org:443/sdk, certs=[[
[
Version: V3
Subject: C=US, CN=vcsa.example.org
Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

Key: Sun RSA public key, 2048 bits
params: null
modulus: ...

2020-09-09 07:08:32.310 WARN - [UI-35ba8eb5-f6bc-4e65-9d70-f6578446796c-ay] [https-jsse-nio-8440-exec-2] c.v.h.r.diagnostics.DiagnosticsService   : Unable to connect to vc b335f0e2-7657-40eb-85cb-42be2eb28a26 with manager 2a98aaa6-fc83-475e-a7ed-7cd34c126e6e's credentials

com.vmware.exception.ConnectTimeout: org.apache.http.conn.ConnectTimeoutException: Connect to vcsa.example.org:443 [vcsa.example.org/192.168.1.10] failed: connect timed out
    at com.vmware.exception.converter.ClientExceptionConverter.convertException(ClientExceptionConverter.java:38)
    at com.vmware.vlsi.util.ExceptionConverterInterceptor.handleException(ExceptionConverterInterceptor.java:30)
    at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:254)
    at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
    ...

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCloud Availability 3.5.x

Cause

This issue can occur when the Lookup Service contains a vCenter Server registration that is not accessible from the Cloud Director Availability On-Premises Appliance as Cloud Director Availability retrieves connection information for the on-premises vCenter Servers by querying the vCenter Server registrations in the Lookup Service.

Resolution

To resolve this issue, ensure that there are no stale vCenter Server registrations in the Lookup Service and that the Cloud Director Availability On-Premises Appliance is able to access each registered vCenter Server.

To check the vCenter Server registrations in the Lookup Service, perform the following steps.

SSH to each vCenter Server and log in as root.
Take note of the instance UUID for each vCenter Server:

grep instanceUuid /etv/vmware-vpx/instance.cfg

If on a separate machine, SSH to the Platform Services Controller and log in as root.
Export the service registration list by running the following command:

/usr/lib/vmidentity/tools/scripts/lstool.py list --url http://localhost:7080/lookupservice/sdk > sso_services

Compare the instance UUIDs from step 2 with the service ID of each vCenter Server registration in the sso_services file to search for stale entries.

If your environment contains stale entries that need to be removed, contact VMware Support and note this Article ID (80744) in the problem description for assistance in cleaning the Lookup Service registrations. For more information, see How to Submit a Support Request.

Additional Information

For more information on viewing Lookup Service registrations, see Process to view the List of Services Registered with Single Sign-On.
For more information on Cloud Director Availability network requirements, see the Deployment Requirements On-Premises section of the Cloud Director Availability documentation .