"Internal server error" is shown and Cloud Resources fail to load in the HTML5 UI in a Multisite environment

Products

VMware Cloud Director

Issue/Introduction

Symptoms:

The Cloud Director instance is configured with Multisite with site associations to other Cloud Director instances.
Cloud Resources pages including Organizations and Organization VDCs pages will not load objects in the HTML5 UI
The UI shows "Internal server error" messages when trying to load Cloud Resources pages
The browser console shows that requests to load objects have failed due to "502 Bad Gateway" errors, for example attempting to load Cloud Reources > Organization VDCs results in the following in the browser console:

Request:

GET https://vcloud.example.com/api/query?type=adminOrgVdc&format=records&pageSize=25&sortAsc=name&links=true
Request Headers:
Accept:application/*+json;version=33.0;multisite=global
Response:
502 Bad Gateway
X-VMWARE-VCLOUD-REQUEST-ID:<REQUEST_UUID>

The response in the browser console shows "SocketTimeoutException" errors for the /api/versions or /api/session public address endpoints of the local site:

com.vmware.vcloud.api.rest.toolkit.exceptions.RestApiException: Internal server error\n\tat
com.vmware.vcloud.multisite.query.QueryExecutor.execute(QueryExecutor.java:109)\n\tat
com.vmware.vcloud.api.rest.multisite.MultisiteVcloudApiFilter.handleQuerySvcRequest(MultisiteVcloudApiFilter.java:143)\n\tat
...
Caused by: javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: SocketTimeoutException invoking https://vcloud.example.com/api/session: connect timed out\n\tat

com.vmware.vcloud.api.rest.toolkit.exceptions.RestApiException: Internal server error\n\tat
com.vmware.vcloud.multisite.query.QueryExecutor.execute(QueryExecutor.java:109)\n\tat
com.vmware.vcloud.api.rest.multisite.MultisiteVcloudApiFilter.handleQuerySvcRequest(MultisiteVcloudApiFilter.java:143)\n\tat
...
Caused by: javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: SocketTimeoutException invoking https://vcloud.example.com/api/versions: connect timed out\n\tat

The /opt/vmware/vcloud-director/logs/vcloud-container-debug.log show errors connecting to the /api/versions or /api/session public address endpoints of the local site when searching for the request ID from the browser logs:

| DEBUG | pool-jetty-69 | QueryExecutor | Internal server error | requestId=<REQUEST_UUID>,request=GET https://vcloud.example.com/api/query,requestTime=1593003126966,remoteAddress=<IP>:57364,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=application/*+json;version 33.0;multisite global
javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: SocketTimeoutException invoking https://vcloud.example.com/api/session: connect timed out

Environment

VMware Cloud Director 10.x

Cause

This issue occurs if the Cloud Director Cells cannot reach the public address of their own site.

Resolution

Ensure that all Cloud Director Cells in an instance can reach the public address endpoints of their own site.
This can be tested on the Cells using Curl:

# curl -k https://vcloud.example.com/api/versions
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<SupportedVersions xmlns="http://www.vmware.com/vcloud/versions" xmlns:ns2="http://www.vmware.com/vcloud/v1.5" xmlns:vmext="http://www.vmware.com/vcloud/extension/v1.5" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:vssd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData" xmlns:common="http://schemas.dmtf.org/wbem/wscim/1/common" xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData" xmlns:vmw="http://www.vmware.com/schema/ovf" xmlns:ovfenv="http://schemas.dmtf.org/ovf/environment/1">
    <VersionInfo deprecated="true">
        <Version>27.0</Version>
        <LoginUrl>https://vcloud.example.com/api/sessions</LoginUrl>
    </VersionInfo>
    <VersionInfo deprecated="true">
        <Version>28.0</Version>
        <LoginUrl>https://vcloud.example.com/api/sessions</LoginUrl>
    </VersionInfo>
    <VersionInfo deprecated="false">
        <Version>29.0</Version>
        <LoginUrl>https://vcloud.example.com/api/sessions</LoginUrl>
    </VersionInfo>
    <VersionInfo deprecated="false">
        <Version>30.0</Version>
        <LoginUrl>https://vcloud.example.com/api/sessions</LoginUrl>
    </VersionInfo>
    <VersionInfo deprecated="false">
        <Version>31.0</Version>
        <LoginUrl>https://vcloud.example.com/api/sessions</LoginUrl>
    </VersionInfo>
    <VersionInfo deprecated="false">
        <Version>32.0</Version>
        <LoginUrl>https://vcloud.example.com/api/sessions</LoginUrl>
    </VersionInfo>
    <VersionInfo deprecated="false">
        <Version>33.0</Version>
        <LoginUrl>https://vcloud.example.com/api/sessions</LoginUrl>
    </VersionInfo>
    <SchemaRoot>https://vcloud.example.com/api/v1.5/schema/</SchemaRoot>
</SupportedVersions>

Workaround:

As a workaround until access to the public endpoint can be provided to the Cells the /etc/hosts file on each Cell can be edited to resolve the public address of the Cloud Director instance to either the IP of Cells themselves or an internal load balancer IP.
For example for the public address vcloud.example.com the following line could be added where <IP> is the Primary IP of the Cell on which we are making the edit:

Edit the Cell's hosts file:

# vi /etc/hosts

At the bottom of the file add the desired IP and FQDN:

<IP> vcloud.example.com

Verify that the Cell can access the public address using these values:

# curl -k https://vcloud.example.com/api/versions