"Unexpected Vcloud director error {{0}}" error when creating replications in VMware Cloud Director Availability 4.x

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Configuring a replication from an On-Premises site to a Cloud site fails with the error:

[ object object ]

• Configuring a replication from a Cloud site to an On-Premises site fails with the error:

Unexpected Vcloud director error {{0}}

• In /opt/vmware/h4/cloud/log/cloud.log on the Cloud Replication Management Appliance, you see similar messages to:

2020-04-13 14:46:05.709 ERROR - [UI-########-####-####-####-########b022-F2] [https-jsse-nio-8443-exec-4] c.v.h.c.c.error.ExceptionAdvisorBase     : A GET request from User@VMware [10.10.10.10] to /inventory/vapps?sort=&offset=0&limit=20&site=VMware failed.
com.vmware.vcloud.client.exception.VcloudException: HTTP Response code: 414
        at com.vmware.vcloud.client.VcloudClient.lambda$defaultErrorDeserializer$0(VcloudClient.java:195)
        at com.vmware.rest.client.AbstractRestClient.convert(AbstractRestClient.java:215)

• In /opt/vmware/h4/cloud/log/request.log on the Cloud Replication Management Appliance, you see similar mesages:

2020-04-14 13:22:33.017 TRACE - [########-####-####-####-########791a] [c4-scheduler-2] c.vmware.rest.client.AbstractRestClient  : Request to GET https://<vcd_fqdn>/api/query?type=adminOrgVdcStorageProfile&format=idrecords&offset=0&pageSize=100&filter=(vdc==########-####-####-####-########4622,vdc==########-####-####-####-########38bd,vdc==02aa907d-98d8-47a5-b39e...
2020-04-14 13:22:33.018  WARN - [########-####-####-####-########791a] [c4-scheduler-2] c.vmware.rest.client.AbstractRestClient  : The HTTP response has no body. Code = 414, reason =

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x
VMware vCloud Availability 3.0.x
VMware vCloud Availability 3.5.x

Cause

The issue occurs because the HTTP request send from Cloud Director Availability to vCloud Director has a header larger than 8KB.

The default supported HTTP header is 8KB for Apache. When the header exceeds this value the query times out, Cloud Director Availability is unable to fetch the resources from vCloud Director and no response is returned.

When limiting the accessible Provider VDCs in Cloud Director Availability, the HTTP header size is determined by the number of Organization VDCs that Cloud Director Availability has access to. The maximum amount of Organization VDCs that a single Cloud Director Availability instance can handle in this configuration is ~164, this number can vary depending on the length of the vCloud Director URL.

Resolution

This is a known issue affecting VMware Cloud Director Availability 4.x.
Currently, there is no resolution.

Workaround:
To work around this issue, modify the accessible Provider VDC setting in the Cloud Director Availability Provider Portal to change how the vCloud Director API request header is built.

1. Navigate to the Cloud Director Availability Provider Portal.
2. Log in with either as root or a vCloud Director System Administrator user.
3. In the left pane, click Configuration.
4. Under Site details, next to Accessible Provider VDCs, click Edit.
5. Select VMware Cloud Director Availability can access all Provider VDCs and click Apply.

If you cannot give Cloud Director Availability access all Provider VDCs because your vCloud Director instance contains Provider VDCs that are backed by vCenter Servers under different Single Sign-On domains or because this would go against the supported configuration limits, you can limit the amount of  Organization VDCs that a single Cloud Director Availability instance handle by deploying more Cloud Director Availability instances on your environment.