Error: Cluster Creation Failed or Cluster creation task do not complete while CSE Kubernetes Cluster creation in vCD

Products

• VMware Cloud Director

Issue/Introduction

• Unable to Create Kubernetes Cluster via Container Service Extension (CSE).

Error: Cluster Creation Failed (OR) No Error, the task doesn't complete

• The following errors are observed in the "/var/log/cloud-final.err" file on the Bootstrap EPHEMERAL-TEMP-VM:

+ echo 'Waiting for kubeconfig to be available...'
Waiting for kubeconfig to be available...
Error: "ClusterName-kubeconfig" not found in namespace "NamespaceName": secrets "<cluster_name>-kubeconfig" not found

 

 

Environment

VMware Cloud Director Container Service Extension (CSE) 4.X/3.X

Cause

Network connectivity issue between VMware Cloud Director and CSE Server.

Resolution

Here is the sequence of steps to diagnose a network issue between VMware Cloud Director and the CSE Server:

• Verify if AVI is used as the load balancer for Kubernetes cluster creation via CSE. If it is, check whether there are sufficient AVI licenses available to create the virtual service.
• If the enough number of AVI Licenses are not available, then creation of virtual services fail.
• Once the required number of AVI licenses are assigned, the creation of virtual services should proceed successfully as expected.
• This can be checked from the Tenant: Network > Edge gateway > Load Balancer > Virtual Services.
• Manually create the virtual service and verify if it remains stable.
• From Ephemeral VM and CSE Server check if we can connect below 3 URLs: 

         curl -k -v https://<VCD_URL>/oauth/provider/token
    curl -k -v https://<VCD_URL>/oauth/provider
    curl -k -v https://<VCD_URL>

• If the URLs do not connect, it indicates a network issue between vCD and CSE. Customers are advised to work with their internal network team to establish the connection from Ephemeral VM and CSE Server to the above said URLs.
• Once the URLs connect successfully from both CSE and the Ephemeral VM, deployment of CSE can proceed.