Workload cluster creation halts with error message "Cluster control Plane is still being initialized, retrying"

Products

• VMware

Issue/Introduction

Symptoms:
You can see errors similar to the ones reported below in the capi-system logs

I0811 11:52:57.843336       1 clusterresourceset_controller.go:236] controllers/ClusterResourceSet "msg"="Applying ClusterResourceSet to cluster" "cluster-name"="mgmt-rp24" "clusterresourceset"="mgmt-rp24-tkr" "namespace"="tkg-system"
E0811 11:53:05.866989       1 machinehealthcheck_controller.go:189] controllers/MachineHealthCheck "msg"="error creating remote cluster cache" "error"="error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster \"default/workload-b\": Get https://10.213.44.248:6443/api?timeout=10s: context deadline exceeded" "cluster"="workload-b"
E0811 11:53:05.868900       1 machinehealthcheck_controller.go:167] controllers/MachineHealthCheck "msg"="Failed to reconcile MachineHealthCheck" "error"="error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster \"default/workload-b\": Get https://10.213.44.248:6443/api?timeout=10s: context deadline exceeded" "cluster"="workload-b"
E0811 11:53:05.869350       1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster \"default/workload-b\": Get https://10.213.44.248:6443/api?timeout=10s: context deadline exceeded" "controller"="machinehealthcheck" "name"="workload-b" "namespace"="default"
E0811 11:53:06.101112       1 clusterresourceset_controller.go:343] controllers/ClusterResourceSet "msg"="failed to apply ClusterResourceSet resource" "error"="failed to create object kapp.k14s.io/v1alpha1, Kind=Config /: no matches for kind \"Config\" in version \"kapp.k14s.io/v1alpha1\"" "cluster-name"="mgmt-rp24" "clusterresourceset"="mgmt-rp24-antrea" "namespace"="tkg-system" "Resource kind"="Secret" "Resource name"="mgmt-rp24-antrea-crs"
E0811 11:53:06.114407       1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create object kapp.k14s.io/v1alpha1, Kind=Config /: no matches for kind \"Config\" in version \"kapp.k14s.io/v1alpha1\"" "controller"="clusterresourceset" "name"="mgmt-rp24-antrea" "namespace"="tkg-system"
I0811 11:53:06.114752       1 clusterresourceset_controller.go:236] controllers/ClusterResourceSet "msg"="Applying ClusterResourceSet to cluster" "cluster-name"="mgmt-rp24" "clusterresourceset"="mgmt-rp24-antrea" "namespace"="tkg-system"
E0811 11:53:15.910083       1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster \"default/workload-b\": Get https://10.213.44.248:6443/api?timeout=10s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "controller"="clusterresourceset" "name"="workload-b-tkg-metadata" "namespace"="default"
I0811 11:53:15.910575       1 clusterresourceset_controller.go:236] controllers/ClusterResourceSet "msg"="Applying ClusterResourceSet to cluster" "cluster-name"="workload-b" "clusterresourceset"="workload-b-tkg-metadata" "namespace"="default"

Environment

VMware Tanzu Kubernetes Grid 1.x

Cause

One of the scenarios where you can encounter this error is when "VSPHERE_CONTROL_PLANE_ENDPOINT" variable is set to an IP that is not reachable.

Resolution

Set VSPHERE_CONTROL_PLANE_ENDPOINT variable to an IP address that is reachable from the machine running tanzu/tkg CLI