Organizations list intermittently empty on the VMware Cloud Director Provider Portal

Products

• VMware Cloud Director

Issue/Introduction

• When browsing the Organizations the list is intermittently empty.

• In /opt/vmware/vcloud-director/logs/vcloud-container-debug.log there are errors similar the errors reported below:

  DATE TIME | ERROR    | pool-jetty-54             | DefaultExceptionMapper         | Internal server exception |
requestId=########-####-####-########73a4,request=GET https://cloud.example.com/cloudapi/1.0.0/orgs,requestTime=17810
08781567,remoteAddress=##.##.##.##:53633,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=a
pplication/json;version 40.0.0-alpha;multisite ########-####-####-########85f9@########-####-####-########91e2:a93..
java.io.IOException: Close SendCallback@7cc3ea2f[PROCESSING][i=null,cb=org.eclipse.jetty.server.HttpChannel$SendCallback@11cc6
f62] in state PROCESSING

  AND

  DATE TIME | ERROR    | pool-jetty-54             | DefaultExceptionMapper         | Internal server exception |
requestId=########-####-####-########73a4,request=GET https://cloud.example.com/cloudapi/1.0.0/orgs,requestTime=17810
08781567,remoteAddress=##.##.##.##:53633,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=a
pplication/json;version 40.0.0-alpha;multisite ########-####-####-########85f9@########-####-####-########91e2:a93..
org.eclipse.jetty.io.EofException: Closed

  AND

  DATE TIME | DEBUG    | pool-76-thread-1          | AssociationRequestExecutor     | Association request failed for System a########-####-####-########85f9@6########-####-####-########91e2 |
javax.ws.rs.ProcessingException: java.net.SocketException: SocketException invoking https://cloud.example.com/cloudapi/1.0.0/orgs?filterEncoded=true&filterEncoded=true&filter=id%21%3Durn%3Avcloud%3Aorg%3########-####-####-########85f9&sortAsc=name&links=true&cursor=eyJvZmZzZX
QiOjAsInBhZ2VTaXplIjoxNX0%3D&pageSize=15: Unexpected end of file from server
[...]
Caused by: java.net.SocketException: SocketException invoking https://cloud.example.com/cloudapi/1.0.0/orgs?filterEncoded=true&filterEncoded=true&filter=id%21%3Durn%3Avcloud%3Aorg%########-####-####-########85f9&sortAsc=name&links=true&cursor=eyJvZmZzZXQiOjAsInBhZ2VTaXplIjoxNX0%3D&pageSize=15: Unexpected end of file from server

Environment

VMware Cloud Director 10.6.x

Cause

This issue occurs when VMware Cloud Director (VCD) Cells cannot successfully perform API calls to their own public VCD endpoint through the Load Balance (LB), due to header or response size limitations on the LB.

Resolution

If using an NSX-T load balancer,

1. Login to the NSX-T Manager UI.
2. Navigate to "Networking > Network Services > Load Balancing > Profiles".
3. Identify the HTTP & HTTP profile in use and ensure the following configuration is in use:
   
   Type: HTTPS
   HTTP Request Header: X-Forward-For
   Idle Timeout: 15 Seconds
   Response Timeout: 60 Seconds
   Request Header Size: 65535
   Response Header Size: 50000

These details are outlined in the VMware Cloud Director Cell Load Balancing Configuration

If third-party load balancers are deployed, access the respective management console to review and update the HTTP/HTTPS profiles to match requirements.