VCFA Kubernetes cluster creation stuck unable to reconcile kubeadm ConfigMap's CoreDNS info

Products

VCF Automation Tanzu Kubernetes Runtime VMware vSphere Kubernetes Service

Issue/Introduction

When creating a vSphere Kubernetes Service (VKS) Kubernetes cluster from VMware Cloud Foundation Automation (VCFA), the cluster remains stuck in Failed state with vSphere Kubernetes service status showing as Not Available and does not progress past the first control plane node. This issue also affects Private AI (PAIS) clusters.

While connected to the Supervisor cluster context, the following symptoms are observed:

The Kubernetes cluster only has one control plane machine created with a corresponding virtual machine that has an IP address successfully assigned:
```
kubectl get machine,vm -o wide -n <namespace>
```

Describing the Kubernetes cluster shows an error similar to the following where values in brackets <> will vary by environment:

kubectl describe cluster <cluster name> -n <namespace>

message: |-      Addon DNS is not ready: unable to reconcile kubeadm ConfigMap's CoreDNS info: unable to retrieve kubeadm Configmap from the guest cluster: failed to get server groups: Get "https://<control plane endpoint IP>:6443/api?timeout=10s": dial tcp <control plane endpoint IP>:6443: connect: connection refused      Addon KubeProxy is not ready: unable to retrieve kube-proxy daemonset from the guest cluster: failed to get server groups: Get "https://<control plane endpoint IP>:6443/api?timeout=10s": dial tcp <control plane endpoint IP>:6443: connect: connection refused

The <control plane endpoint IP> corresponds to the Control Plane Endpoint at port 6443 automatically assigned to the cluster on creation.

This can be verified with the following command:

kubectl get cluster -o yaml <cluster name>  -n <namespace> | grep -i endpoint -A2

You are unable to SSH into the control plane node despite that an IP address is assigned to it.

Kubectl commands using the Kubernetes cluster's kubeconfig return any of the following errors:

couldn't get server api group list

Get "https://<control plane endpoint IP>:6443/api/v1/namespaces/default/configmaps/<kubernetes cluster>-kapp-controller.app.apps.k14s.io": dial tcp <control plane endpoint IP>:6443: connect: connection refused

Get "https://<control plane endpoint IP>:6443/api/v1/namespaces/default/configmaps/<kubernetes cluster>-kapp-controller.app.apps.k14s.io": tls: failed to verify certificate: x509: certificate is valid for <IP A>, <IP B>, <IP C>, <localhost>, not <control plane endpoint IP>

The Kubernetes Cluster's control plane endpoint external IP address is a duplicate of an existing kubernetes service in the Supervisor cluster:

kubectl get services -A | grep "<control plane endpoint IP>"

<namespace>    <kubernetes cluster>   LoadBalancer   <internal IP>   <control plane endpoint IP>   6443/TCP                       

kube-system    kube-apiserver-lb-svc     LoadBalancer   <internal IP>   <control plane endpoint IP>

In VMware Cloud Foundation Automation | Provider Management under Networking - External IP Blocks, one or more external IP blocks is using the same CIDR as the above external IP addresses of the kubernetes services in the Supervisor cluster.

In the vSphere web UI, the Supervisor cluster's Configure - Network - Workload Networks External IP Block is the same CIDR as the above External IP Block noted in VCFA | Provider Management.

The VCFA External IP Block and Supervisor cluster are in the same Region.

Environment

vSphere Supervisor 9.X

VMware Cloud Foundation Automation 9.1

Cause

The VCFA External IP block used by the same Region as Supervisor cluster is using the same duplicate CIDR as the Supervisor cluster's Workload Networks External IP block.

The control plane endpoint for the created Kubernetes cluster uses the CIDR from the VCFA external IP Block defined for the same region.

This creates duplicate IP conflicts where the control plane endpoint of any Kubernetes clusters can be the same as the external IP address of a Supervisor cluster's service.

The control plane endpoint of a Kubernetes cluster is used to contact the kube-apiserver of the Kubernetes cluster which is needed for cluster management including initial deployment.

Kubernetes cluster creation will fail if the system is unable to create or reach the kube-apiserver container process expected to be deployed in the new Kubernetes cluster.

No containers will be able to run in the new Kubernetes cluster.

Resolution

In VCFA | Provider Management, correct the existing external IP block so that does not overlap or duplicate the external IP block used by the Supervisor cluster's Workload Network.

Clean up all failed Kubernetes cluster deployments in the affected namespace and region from the VCFA web UI.
Login to VCFA | Provider Management and correct the existing external IP block in the same region:
- This can be done by editing the existing external IP block, or by creating a new external IP block for the same Region then removing the problematic external IP block.

So long as there are no other duplicate or overlapping CIDR blocks or networking misconfiguration that would result in similar behavior, the next kubernetes cluster deployment should use the corrected external IP block to assign its control plane endpoint and create successfully.

Additional Information

The Supervisor Cluster's Workload Networks - Services CIDR decides the internal IPs for the Kubernetes services on the Supervisor cluster.

The Supervisor Cluster's Workload Networks - External IP Blocks defines the external IPs for the Kubernetes services on the Supervisor cluster.

The external IP for a Kubernetes Cluster's Control Plane Endpoint pulls from the VCFA external IP block defined for the same Region that the Supervisor cluster is associated with.