Global Roles Cannot be Created or Edited in the Cloud Director HTML5 UI

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• You cannot select rights when creating a Role
• You cannot see existing rights when editing a Role
• Viewing the Console Browser Logs (F12), you see API Calls failing when selecting roles
• Cloud Director API Requests using the /cloudapi/ API fail with a TCP RST (ERR_CONNECTION_RESET)
• "unexpected character at line 1 column 1 of the JSON data" error message observed in the Cloud Director HTML5 UI

Environment

Cloud Director

Cause

This issue occurs when you are connecting to Cloud Director via a Load Balancer which has limitations set to the Header Size or Header Count.
The Header Size or Header Count sent by Cloud Director to the Load Balancer exceed these values, and the Load Balancer resets the connection causing the API Call to fail.

The new Cloud API (cloudapi) is based on OpenAPI which places links in the headers making the Payload and Header Count substantially larger than the traditional /api/ API.

Resolution

To resolve the issue, you can either a) Edit the Load Balancer Config, or b) Edit the Cloud Director Config
 

Edit the Load Balancer Config

Ensure that the values set on the Load Balancer are sufficient to receive the API Request responses from Cloud Director.

• If you are using an NSX Load Balancer please update the configuration using the NSX API
  • PUT https://<NSX-Manager>/api/4.0/edges/<Edge-ID>/systemcontrol/config
     
    <systemControl>
        <property>lb.global.tune.http.maxhdr=1024</property>  -- 256 may also be valid, but 1024 is a recommended value
        <property>lb.global.tune.bufsize=65536</property>
    </systemControl>

• If you are using an F5 Load Balancer, the below values have previously been noted to be sufficient.
  • Maximum Header Size:  65536
  • Maximum Header Count: 256
• If you are using HA Proxy, the below values have previously been noted to be sufficient
  
  • tune.http.maxhdr=1024
  • tune.bufsize=65536 

Note: Whilst the above values are currently sufficient, this may not always be the case. VMware Engineering may look to update the Headers section of various cloudapi requests in future releases of Cloud Director which may require the existing Load Balancer settings to be increased
 

Edit the Cloud Director Config

You can also amend Cloud Director to reduce the overall size of Requests and Headers sent to Load Balancers via the cell management-tool manage-config subcommand.
See Updating Application Configuration Settings in VMware Cloud Director for more details.

• restapi.queryservice.maxPageSize
• vcloud.http.responseHeaderMax

Example

• Set restapi.queryservice.maxPageSize to reduce the amount of elements per page to 64.
  • /opt/vmware/vcloud-director/bin/cell-management-tool  manage-config -n restapi.queryservice.maxPageSize -v 64