Management cluster creation failing with error "open /tmp/k8s-webhook-server/serving-certs/tls.crt: no such file or directory" in ako-operator-controller-manager

Products

• Tanzu Kubernetes Runtime
• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x

Issue/Introduction

When creating tanzu management cluster the bootstrap cluster is created successfully but the management cluster creation is stuck with the below message repeating continuously.

control plane is not available yet, retrying
control plane is not available yet, retrying
control plane is not available yet, retrying

After running kubectl from inside the bootstrap kind cluster we observe that the ako-operator-controller-manager pod is in CrashLoopBackOff state

kubectl get pods -A | grep -i ako
tkg-system       ako-operator-controller-manager-56b858b79d-r57gp     0/1   CrashLoopBackOff  5 (2m10s ago)  5m29s

From the logs of the ako-operator-controller-manager pod the following error message is observed

2025-01-25T18:08:49.370Z	ERROR	controller-runtime.source.EventHandler	failed to get informer from cache	{"error": "Timeout: failed waiting for *v1beta1.Cluster Informer to sync"}
<----------------Truncated for brevity------------------>
2025-01-25T18:08:49.372Z	ERROR	controller-runtime.source.EventHandler	failed to get informer from cache	{"error": "Timeout: failed waiting for *v1.Service Informer to sync"}
<----------------Truncated for brevity------------------>
2025-01-25T18:08:49.373Z	ERROR	controller-runtime.source.EventHandler	failed to get informer from cache	{"error": "Timeout: failed waiting for *v1beta1.Machine Informer to sync"}
<----------------Truncated for brevity------------------>
2025-01-25T18:08:49.373Z	ERROR	controller-runtime.source.EventHandler	failed to get informer from cache	{"error": "Timeout: failed waiting for *v1.Secret Informer to sync"}
<----------------Truncated for brevity------------------>	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:56
2025-01-25T18:08:49.373Z	INFO	Stopping and waiting for webhooks
2025-01-25T18:08:49.373Z	INFO	Wait completed, proceeding to shutdown the manager
2025-01-25T18:08:49.373Z	ERROR	problem running manager	{"error": "open /tmp/k8s-webhook-server/serving-certs/tls.crt: no such file or directory"}
main.main
	/workspace/main.go:102
runtime.main
	/usr/local/go/src/runtime/proc.go:250

Cause

From the management cluster creation logs we can see that the CLUSTER_TOPOLOGY configuration is being set to false. Because the value of CLUSTER_TOPOLOGY is false the package based cluster class creation is disabled. Due to this tanzu-addons-controller-manager and kapp-controller are not installed on the bootstrap cluster. This causes the failure in creation of secrets necessary for ako-operator-controller-manager to function correctly.

Checking if VSPHERE_CONTROL_PLANE_ENDPOINT  is already in use
SourceFilePath: /home/worker/.kube-tkg/config, ContextName: mgmt-avi-admin@mgmt-avi
Namespace: default, Cluster Name: mgmt-avi

Setting up management cluster...
Validating configuration...
CLUSTER_TOPOLOGY configuration already set to "false"
Using infrastructure provider vsphere:v1.8.8

This can be confirmed by reviewing if the package-based-cc feature is enabled under ~/.config/tanzu/config.yaml file. As shown in the example below, the feature should be true for a successful cluster creation. If the parameter is missing from the configuration file or set to false the management cluster creation will fail.

grep package-based-cc config-tanzu-config.yaml
            package-based-cc: "true"

Resolution

To resolve this issue set package-based-cc feature to true using tanzu cli and confirm it's reflected in ~/.config/tanzu/config.yaml

tanzu config set features.management-cluster.package-based-cc true

# To Confirm
grep package-based-cc ~/.config/tanzu/config.yaml
            package-based-cc: "true"