"json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)" error causes a migration to fail in VMware NSX Migration for VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Attempting a migration of an NSX-V backed Org VDC to an NSX-T backed Org VDC fails when using VMware NSX Migration for VMware Cloud Director.
• The migration attempt fails with an error in the VCD-NSX-Migrator-Main-<date>.log similar to the following:

[vcdConfigureEdgeGatewayServices]:[configureServices]:84 [ERROR] [ExampleOrgVdc] | Traceback (most recent call last):
  File "src\core\vcd\vcdConfigureEdgeGatewayServices.py", line 71, in configureServices
  File "src\core\vcd\vcdValidations.py", line 150, in wrapped
  File "src\core\vcd\vcdValidations.py", line 90, in inner
  File "src\core\vcd\vcdValidations.py", line 80, in inner
  File "src\core\vcd\vcdConfigureEdgeGatewayServices.py", line 793, in configureTargetNAT
  File "src\core\vcd\vcdValidations.py", line 53, in inner
  File "src\core\vcd\vcdConfigureEdgeGatewayServices.py", line 664, in getApplicationPortProfiles
  File "requests\models.py", line 910, in json
  File "json\__init__.py", line 357, in loads
  File "json\decoder.py", line 337, in decode
  File "json\decoder.py", line 355, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
...
[vcdNSXMigrator]:[run]:991 [ERROR] [ExampleOrgVdc] | VCD V2T Migration Tool Failed, Please Check the logs for the exceptions
Traceback (most recent call last):
  File "src\vcdNSXMigrator.py", line 893, in run
  File "src\commonUtils\threadUtils.py", line 170, in waitForThreadToComplete
Exception: VCD V2T Migration Tool Failed, Please Check the logs for the exceptions

• The VCD-NSX-Migrator-Main-<date>.log shows a 502 - Bad Gateway response to a GET /cloudapi/1.0.0/applicationPortProfiles API call from the VMware NSX Migration for VMware Cloud Director to the Cloud Director:

[connectionpool]:[_make_request]:452 [DEBUG] [ExampleOrgVdc] | https://vcloud.example.com:443 "GET /cloudapi/1.0.0/applicationPortProfiles?page=1&pageSize=75&filter=_context==urn:vcloud:nsxtmanager:<nsxt-manager-uuid>&sortAsc=name HTTP/1.1" 502 None

• Performing the same API call to the Cloud Director outside of VMware NSX Migration for VMware Cloud Director also results in the same 502 - Bad Gateway response.
• Cloud Director is located behind a Load Balancer or other network device.

Environment

VMware Cloud Director 10.x

Cause

This issue occurs when you are connecting to Cloud Director via a Load Balancer or other network device which has limitations set to the Header Size, Header Count or Payload.
If the Header Size, Header Count, or Payload sent by Cloud Director to the Load Balancer exceeds these values, the Load Balancer resets the connection causing the API call to fail.
The Cloud Director OpenAPI (/cloudapi) is based on OpenAPI which places links in the headers making the Payload and Header Count substantially larger than the traditional Cloud Director API (/api).

Resolution

To resolve the issue ensure that the Load Balancer's maximum header size and count is sufficient to allow the API response from Cloud Director to succeed.
Please see the relevant documentation for the specific Load Balancer in place to achieve this.

For an NSX-T Load Balancer we may set values such as the following:

• Request Header Size: 65535
• Request Body Size: 52428800
• Response Header Size: 50000