Error: Cluster Creation Fails or the task remains in "Pending" state when creating CSE Kubernetes Cluster in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• Unable to Create Kubernetes Cluster via Container Service Extension (CSE).

Error: Cluster Creation Failed (OR) No Error, the task doesn't complete

• The following errors are observed in the "/var/log/cloud-final.err" file on the Bootstrap EPHEMERAL-TEMP-VM:

+ echo 'Waiting for kubeconfig to be available...'
Waiting for kubeconfig to be available...
Error: "ClusterName-kubeconfig" not found in namespace "NamespaceName": secrets "<cluster_name>-kubeconfig" not found

• When cluster creation task is stuck in “Pending” state, following errors are observed in "/root/cse.log" on the CSE VM

Connecting to [VMware_Cloud_Director_URL]

failed to authenticate using refresh token

panic: error logging into VCD: [unable to get swagger client from secrets: [unable to get bearer token from secrets: [failed to set authorization header: [Post "<VMware_Cloud_Director_URL>": dial tcp xx.xx.xx.xx:xxx: connect: connection timed out]

Environment

Container Service Extension 4.X/3.X

VMware Cloud Director 10.5.x

Cause

Network connectivity issue between VMware Cloud Director and CSE Server.

Resolution

Here is the sequence of steps to diagnose a network issue between VMware Cloud Director and the CSE Server:

• Verify if AVI is used as the load balancer for Kubernetes cluster creation via CSE. If it is, check whether there are sufficient AVI licenses available to create the virtual service.
• If the enough number of AVI Licenses are not available, then creation of virtual services fail.
• Once the required number of AVI licenses are assigned, the creation of virtual services should proceed successfully as expected.
• This can be checked from the Tenant: Network > Edge gateway > Load Balancer > Virtual Services.
• Manually create the virtual service and verify if it remains stable.
• From Ephemeral VM and CSE Server check if we can connect below 3 URLs: 

         curl -k -v https://<VCD_URL>/oauth/provider/token
    curl -k -v https://<VCD_URL>/oauth/provider
    curl -k -v https://<VCD_URL>

• If the URLs do not connect, it indicates a network issue between vCD and CSE. Customers are advised to work with their internal network team to establish the connection from Ephemeral VM and CSE Server to the above said URLs.
• Once the URLs connect successfully from both CSE and the Ephemeral VM, deployment of CSE can proceed.