When creating tanzu management cluster the bootstrap cluster is created successfully but the management cluster creation is stuck with the below message repeating continuously.
control plane is not available yet, retrying
control plane is not available yet, retrying
control plane is not available yet, retrying
After running kubectl from inside the bootstrap kind cluster we observe that the ako-operator-controller-manager
pod is in CrashLoopBackOff
state
kubectl get pods -A | grep -i ako
tkg-system ako-operator-controller-manager-56b858b79d-r57gp 0/1 CrashLoopBackOff 5 (2m10s ago) 5m29s
From the logs of the ako-operator-controller-manager
pod the following error message is observed
2025-01-25T18:08:49.370Z ERROR controller-runtime.source.EventHandler failed to get informer from cache {"error": "Timeout: failed waiting for *v1beta1.Cluster Informer to sync"}
<----------------Truncated for brevity------------------>
2025-01-25T18:08:49.372Z ERROR controller-runtime.source.EventHandler failed to get informer from cache {"error": "Timeout: failed waiting for *v1.Service Informer to sync"}
<----------------Truncated for brevity------------------>
2025-01-25T18:08:49.373Z ERROR controller-runtime.source.EventHandler failed to get informer from cache {"error": "Timeout: failed waiting for *v1beta1.Machine Informer to sync"}
<----------------Truncated for brevity------------------>
2025-01-25T18:08:49.373Z ERROR controller-runtime.source.EventHandler failed to get informer from cache {"error": "Timeout: failed waiting for *v1.Secret Informer to sync"}
<----------------Truncated for brevity------------------> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/source/kind.go:56
2025-01-25T18:08:49.373Z INFO Stopping and waiting for webhooks
2025-01-25T18:08:49.373Z INFO Wait completed, proceeding to shutdown the manager
2025-01-25T18:08:49.373Z ERROR problem running manager {"error": "open /tmp/k8s-webhook-server/serving-certs/tls.crt: no such file or directory"}
main.main
/workspace/main.go:102
runtime.main
/usr/local/go/src/runtime/proc.go:250
From the management cluster creation logs we can see that the CLUSTER_TOPOLOGY
configuration is being set to false
. Because the value of CLUSTER_TOPOLOGY
is false
the package based cluster class creation is disabled. Due to this tanzu-addons-controller-manager
and kapp-controller
are not installed on the bootstrap cluster. This causes the failure in creation of secrets necessary for ako-operator-controller-manager
to function correctly.
Checking if VSPHERE_CONTROL_PLANE_ENDPOINT is already in use
SourceFilePath: /home/worker/.kube-tkg/config, ContextName: mgmt-avi-admin@mgmt-avi
Namespace: default, Cluster Name: mgmt-avi
Setting up management cluster...
Validating configuration...
CLUSTER_TOPOLOGY configuration already set to "false"
Using infrastructure provider vsphere:v1.8.8
This can be confirmed by reviewing if the package-based-cc
feature is enabled under ~/.config/tanzu/config.yaml
file. As shown in the example below, the feature should be true for a successful cluster creation. If the parameter is missing from the configuration file or set to false
the management cluster creation will fail.
grep package-based-cc config-tanzu-config.yaml
package-based-cc: "true"
To resolve this issue set package-based-cc
feature to true using tanzu cli and confirm it's reflected in ~/.config/tanzu/config.yaml
tanzu config set features.management-cluster.package-based-cc true
# To Confirm
grep package-based-cc ~/.config/tanzu/config.yaml
package-based-cc: "true"