Error: "secrets "<cluster_name>-kubeconfig" not found" or the task remains in "Pending" state when creating CSE Kubernetes Cluster

Products

• VMware Cloud Director

Issue/Introduction

• When attempting to deploy a Tanzu Kubernetes cluster using VMware Cloud Director Container Service Extension (CSE), the deployment task remains in a "Pending" state and eventually fails.
• Unable to Create Kubernetes Cluster via CSE.
  
  Error: Cluster Creation Failed (OR) No Error, the task doesn't complete
  
  
• The following errors are observed in the "/var/log/cloud-final.err" file on the Bootstrap EPHEMERAL-TEMP-VM:
  
  + echo 'Waiting for kubeconfig to be available ... '
Waiting for kubeconfig to be available ...
Error: "ClusterName-kubeconfig" not found in namespace "NamespaceName": secrets "<cluster_name>-kubeconfig" not found
  
  
• When cluster creation task is stuck in “Pending” state, following errors are observed in "/root/cse.log" on the CSE VM:
  

  Connecting to [VMware_Cloud_Director_URL]

  failed to authenticate using refresh token

  panic: error logging into VCD: [unable to get swagger client from secrets: [unable to get bearer token from secrets: [failed
set authorization header: [Post "<VMware_Cloud_Director_URL>": dial tcp xx. xx. xx. xx:xxx: connect: connection timed out]

• You can also see the below errors from CAPVCD logs:
  
  [error occurred during the execution of cluster creation script for [<cluster-name>
(urn:vcloud: entity: vmware: capvcdCluster:########-####-####-####-############)]: [error while bootstrapping the machine
[<cluster-name>/EPHEMERAL-TEMP-VM]; timeout for post customization phase
[guestinfo.cloudinit.target.cluster.get.kubeconfig.status]; this usually means that the Avi Load-Balancer is not getting
created; check if virtual service for the cluster is created, if not, please check your Avi configuration]]

Environment

• VMware Cloud Director Container Service Extension 4.x
• VMware Cloud Director 10.x

Cause

The deployment process requires the Ephemeral bootstrap VM to authenticate with the VMware Cloud Director API to create the kubeconfig secret. If the VM cannot reach the VCD public API endpoint, the secret cannot be created, causing the deployment to time out.

Resolution

To resolve this issue, restore network connectivity between the Ephemeral bootstrap VM and the VCD public API endpoint. The Ephemeral VM requires outbound HTTPS (Port 443) access to the VCD public FQDN.

Here is the sequence of steps to diagnose a network issue between the Ephemeral VM and VCD.

Network Connection Verification

1. From the Ephemeral VM:
   1. Verify the VCD public API endpoint can be resolved.
      
      nslookup <VCD_PUBLIC_URL>
      
      If the name does not resolve, verify the DNS configuration in the environment.
      
      
   2. Verify successful endpoint connectivity to the VCD public API endpoint.
      
      curl -k -v https://<VCD_PUBLIC_URL>
      curl -k -v https://<VCD_PUBLIC_URL>/api/versions
      curl -k -v https://<VCD_PUBLIC_URL>/oauth/provider
      curl -k -v https://<VCD_PUBLIC_URL>/oauth/provider/token
      curl -k -v https://<VCD_PUBLIC_URL>/cloudapi/1.0.0/sessions
      
      If the URLs do not connect, it indicates a network issue between VCD and the Ephemeral VM. Customers are advised to work with their internal network team to establish the connection from Ephemeral VM and CSE Server to the above said URLs. Once the URLs connect successfully from both CSE and the Ephemeral VM, deployment of the Kubernetes Cluster can proceed.

Verify Firewall Rules: 

• Audit Distributed Firewall (DFW) rules and physical firewalls to ensure traffic to the VCD API gateway is not blocked.

AVI Verification

• If VMware Avi Load Balancer is used as the load balancer for Kubernetes cluster creation via CSE, confirm that there are sufficient AVI licenses available to create the virtual service. If enough AVI Licenses are not available, then creation of virtual services fail. 
  • The Virtual Servcies status can be checked from the Tenant UI under  Network > Edge gateway > Load Balancer > Virtual Services.

Additional Information

Network and DNS Requirements for CSE Cluster Deployment in VMware Cloud Director 10.x

Prerequisites for installing, configuring and upgrading CSE