"ERROR: Unexpected token "<", "<!DOCTYPE" .. is not valid JSON" when Viewing Edge Gateway NAT Rules

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• You cannot view NAT rules on an Edge Gateway where there are a large number of rules configured
• Viewing the Console Browser Logs (F12), you see API Calls failing when viewing NAT rules
• Cloud Director API Requests using the /cloudapi/ API fail with a TCP RST (ERR_CONNECTION_RESET)
• Cloud Director API Requests using the /cloudapi/ API fail with a 502 (BAD GATEWAY)
• "ERROR: Unexpected token "<", "<!DOCTYPE" .. is not valid JSON" error message observed in the Cloud Director UI

Environment

VMware Cloud Director

Cause

This issue occurs when you are connecting to Cloud Director via a Load Balancer which has limitations set to the Header Size, Header Count, or Response Size that do not meet the recommended settings outlined in the documentation.

VMware Cloud Director Cell Load Balancing Configuration

When these settings are below the recommended configuration, a response sent by the VMware Cloud Director API can exceed them and the Load Balancer resets the connection causing the API call to fail.

The Cloud API (cloudapi) is based on OpenAPI which places links in the headers making the Payload and Header Count substantially larger than default HTTPS profiles on Load Balancers generally allow.

Resolution

Configure the Load Balancer to meet the recommended settings outlined in the documentation.

VMware Cloud Director Cell Load Balancing Configuration

Workaround:

If it is not possible to reconfigure the Load Balancer, you can workaround this issue by limiting the Query Service Max Page Size. However, this will impact page load times across the entire VMware Cloud Director environment. See Updating Application Configuration Settings in VMware Cloud Director for more details.

• To set restapi.queryservice.maxPageSize to reduce the amount of elements in a page to 64 from the default of 128, run the following command on any cell:

/opt/vmware/vcloud-director/bin/cell-management-tool  manage-config -n restapi.queryservice.maxPageSize -v 64

• This command does not require a restart, and only needs to be run on one cell in the environment. You can verify the setting with the following command:

/opt/vmware/vcloud-director/bin/cell-management-tool  manage-config -n restapi.queryservice.maxPageSize -l