"Error creating VCD client to reconcile Cluster ... net/http: TLS handshake timeout" failure when attempting to create a Kubernetes Cluster using Cloud Director Container Service Extension 4.x

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Attempting to create a Kubernetes Cluster using Cloud Director Container Service Extension 4.x fails.
• The CAPVCD logs on the Kubernetes Cluster show errors of the form:

ERROR   Reconciler error        {"controller": "vcdcluster", "controllerGroup": "infrastructure.cluster.x-k8s.io", "controllerKind": "VCDCluster", "vCDCluster": {"name":"mycluster","namespace":"mycluster-ns"}, "namespace": "mycluster-ns", "name": "mycluster", "reconcileID": "<RECONCILE_UUID>", "error": "Error creating VCD client to reconcile Cluster [mycluster] infrastructure: unable to get swagger client from secrets: [unable to get bearer token from secrets: [failed to set authorization header: [Post \"https://vcloud.example.com/oauth/provider/token\": net/http: TLS handshake timeout]]]"

Environment

VMware Cloud Director 10.x

Cause

During the Kubernetes Cluster deployment the Bootstrap VM, EPHEMERAL_TEMP_VM, will attempt to generate an Authorization Token for the Cloud Director instance in order to perform cluster creation steps.
This issue can occur if the EPHEMERAL_TEMP_VM cannot reach the Cloud Director API public address to generate the Authorization Token due to an MTU configuration issue.

Resolution

To confirm if there is an issue with the MTU perform the following steps:

1. Locate the EPHEMERAL_TEMP_VM in the vApp created for the Kubernetes Cluster in the Tenant UI of Cloud Director.
2. Click on the VM to view its Details and open the Guest OS Customization section.
3. Click the Edit option and note the auto generated password which is present under Specify password.
4. Open a VM Console to the EPHEMERAL_TEMP_VM and login using the root user and the password from step 3 above.
5. Test the connectivity from the EPHEMERAL_TEMP_VM to the Cloud Director public address, for example by using ping with a packet size of 1600:

ping vcloud.example.com -s 1600

6. If there is packet loss this could indicate an MTU configuration issue which will stop the EPHEMERAL_TEMP_VM from successfully communicating with the Cloud Director public address.

To resolve the issues with the Kubernetes cluster creation ensure that the MTU configuration within the environment is sufficient to allow communication from the Organization VDC Network where the Kubernetes Cluster vApp VMs are located to the Cloud Director public address and the internet.