Error: Cluster Creation Fails or the task remains in "Pending" state when creating CSE Kubernetes Cluster

Products

• VMware Cloud Director

Issue/Introduction

• Unable to Create Kubernetes Cluster via Container Service Extension (CSE).

  Error: Cluster Creation Failed (OR) No Error, the task doesn't complete

• The following errors are observed in the "/var/log/cloud-final.err" file on the Bootstrap EPHEMERAL-TEMP-VM:

  + echo 'Waiting for kubeconfig to be available...'
  Waiting for kubeconfig to be available...
  Error: "ClusterName-kubeconfig" not found in namespace "NamespaceName": secrets "<cluster_name>-kubeconfig" not found

• When cluster creation task is stuck in “Pending” state, following errors are observed in "/root/cse.log" on the CSE VM

  Connecting to [VMware_Cloud_Director_URL]
  
  failed to authenticate using refresh token
  
  panic: error logging into VCD: [unable to get swagger client from secrets: [unable to get bearer token from secrets: [failed to set authorization header: [Post "<VMware_Cloud_Director_URL>": dial tcp xx.xx.xx.xx:xxx: connect: connection timed out]

  
  
  
• You can also see the below errors from CAPVCD logs:

  [error occurred during the execution of cluster creation script for [<cluster-name>(urn:vcloud:entity:vmware:capvcdCluster:########-####-####-####-############)]: [error while bootstrapping the machine [<cluster-name>/EPHEMERAL-TEMP-VM]; timeout for post customization phase [guestinfo.cloudinit.target.cluster.get.kubeconfig.status]; this usually means that the Avi Load-Balancer is not getting created; check if virtual service for the cluster is created, if not, please check your Avi configuration]]

Environment

VMware Cloud Director Container Service Extension 4.x
VMware Cloud Director 10.5.x

Cause

This issue may be caused because of network connectivity problem between VMware Cloud Director(VCD) and CSE Server or even if there is network connectivity issue between Ephemeral VM, CSE Server and VCD. 

Resolution

Here is the sequence of steps to diagnose a network issue between VMware Cloud Director and the CSE Server:

• Verify if AVI is used as the load balancer for Kubernetes cluster creation via CSE. If it is, check whether there are sufficient AVI licenses available to create the virtual service.
• If the enough number of AVI Licenses are not available, then creation of virtual services fail.
• Once the required number of AVI licenses are assigned, the creation of virtual services should proceed successfully as expected.
• This can be checked from the Tenant: Network > Edge gateway > Load Balancer > Virtual Services.
• Manually create the virtual service and verify if it remains stable.
• From Ephemeral VM and CSE Server check if we can connect below 3 URLs:
  
  curl -k -v https://<VCD_URL>/oauth/provider/token
curl -k -v https://<VCD_URL>/oauth/provider
curl -k -v https://<VCD_URL>
curl -k -v https://<VCD_URL>/api/versions
curl -k -v https://<VCD_URL>/cloudapi/1.0.0/sessions

• If the URLs do not connect, it indicates a network issue between VCD and CSE. Customers are advised to work with their internal network team to establish the connection from Ephemeral VM and CSE Server to the above said URLs.
• Once the URLs connect successfully from both CSE and the Ephemeral VM, deployment of CSE can proceed.
• For further prerequisites regarding network connectivity between VCD and CSE please have a look at this document: Prerequisites for installing, configuring and upgrading CSE