Error: "timeout for post customization phase" and "this usually means that the Avi Load-Balancer is not getting created" during creation of Kubernetes cluster.

Products

• VMware Cloud Director

Issue/Introduction

• Creation of Kubernetes cluster fails in VMware Container Service Extension (CSE).
• The following event is logged as the reason:

Detailed Error: [error while bootstrapping the machine [ninjatkgmads/EPHEMERAL-TEMP-VM]; timeout for post customization phase [guestinfo.cloudinit.target.cluster.get.kubeconfig.status]; this usually means that the Avi Load-Balancer is not getting created; check if virtual service for the cluster is created, if not, please check your Avi configuration] during cluster creation.

• You have validated the resolution from KB "Error: Cluster Creation Fails or the task remains in "Pending" state when creating CSE Kubernetes Cluster"  yet issues persist.

Environment

VMware Cloud Director 10.6.1.1

VMware Container Service Extension 4.2.3

Cause

This issue will occur if the CSE Server Component settings found in the CSE Management UI of the Provider portal are set to incorrect versions.

Resolution

1. Login to the Cloud Director Provider UI as a System Administrator
2. Navigate to "Kubernetes Container Clusters -> CSE Management -> Server Details"
3. Review the values set in the "Component Versions"  section. Cross reference the values set with the version recommendations from Upgrade Kubernetes Components in VMware Cloud Director Container Service Extension Clusters. for your installed version of CSE.
   
   Note: If you click the "UPDATE SERVER" button on the Server Details page and then select "Update Configuration -> Next" it will tell you what the recommended default value for each version section is in the text beside each textbox.
   
   
4. To update the versions:
   1. Click the "UPDATE SERVER" button.
   2. Select "Update Configuration -> Next".
   3. Modify any values required for  CAPVCD, Cloud Provider Interface (CPI), Container Storage Interface (CSI) and RDE Projector Versions 
   4. Click the "Submit Changes" button.
      
      
5. Access the CSE server with SSH or console and restart the CSE service to apply the change.
   
   systemctl restart cse.service
   
   
6. Deploy a new Kubernetes cluster to confirm the issue is resolved.