"timeout for post customization phase [guestinfo.cloudinit.target.cluster.get.kubeconfig.status]] during cluster creation" failure when attempting to create a Kubernetes Cluster using Cloud Director Container Service Extension 4.x

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Attempting to create a Kubernetes Cluster using Cloud Director Container Service Extension 4.x fails.
• An Ephemeral Temp VM is created but the Load Balancer Virtual Service and Load Balancer Pool for the Kubernetes Cluster are not created.
• The Kubernetes Cluster shows an Error Status in the Cloud Director UI, Kubernetes Container Clusters Plug-in.
• The Events on the Kubernetes Cluster show a ScriptExecutionTimeout error with the following detail:

[error while bootstrapping the machine [my-example-cluster/EPHEMERAL_TEMP_VM]; timeout for post customization phase [guestinfo.cloudinit.target.cluster.get.kubeconfig.status]] during cluster creation

Environment

VMware Cloud Director 10.x

Cause

During Kubernetes Cluster creation the Ephemeral Temp VM will make requests to the Cloud Director API at the Cloud Director Public Address.
This issue occurs if the Ephemeral Temp VM cannot access the Cloud Director API at the Cloud Director public address to perform these requests.

Resolution

Ensure that the Routed Organization Network chosen during Kubernetes Cluster creation has internet connectivity to allow cluster VMs to download packages during configuration and access to the Cloud Director Public Address to perform API calls.
For more information see the VMware Cloud Director Container Service Extension Product Documentation on Organization Virtual Data Center Prerequisites for Kubernetes Cluster Deployment.

To confirm if there is an issue with the access to the Cloud Director API perform the following steps:

1. Locate the EPHEMERAL_TEMP_VM in the vApp created for the Kubernetes Cluster in the Tenant UI of Cloud Director.
2. Click on the VM to view its Details and open the Guest OS Customization section.
3. Click the Edit option and note the auto generated password which is present under Specify password.
4. Open a VM Console to the EPHEMERAL_TEMP_VM and login using the root user and the password from step 3 above (the default keyboard language for the Console will be US).
5. Test the connectivity from the EPHEMERAL_TEMP_VM to the Cloud Director public address using Curl:

curl -k -v https://vcloud.example.com/api/versions

6. If the request to the Cloud Director public address does not complete and return XML with the API versions supported by this VCD instance then the API is not reachable from this network and the cluster creation will fail.