VKS workload Cluster Deployment Stuck with Error "Addon DNS is not ready"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When deploying new workload cluster in Supervisor environment, the deployment process stalls. While the machine object appears to be created successfully, the corresponding virtualmachine remains in a PoweredOff state.
  
  

  kubectl get vm,machine -n <affected cluster namespace> | grep  <affected cluster-name>

  NAME                                                               POWER-STATE   CLASS        IMAGE                   PRIMARY-IP4   AGE
virtualmachine.vmoperator.vmware.com/ <control plane node 1>      PoweredOff    <vm-class>  vmi-#########                   ##m

NAME                                                               CLUSTER           NODE NAME   PROVIDER ID   READY   AVAILABLE   UP-TO-DATE   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE   PAUSED   PHASE          AGE    VERSION
machine.cluster.x-k8s.io/ <control plane node 1>                   < cluster-name>                              False   False       True                                                False    Provisioning   ##m    v1.34.1+vmware.

• Describing the cluster show's timeouts related to Addon DNS.
  
  kubectl  describe  <affected cluster-name> -n <affected cluster namespace>

Message: Addon DNS is not ready: unable to reconcile kubeadm ConfigMap's CoreDNS info: unable to retrieve kubeadm Configmap from the guest cluster: failed to get server groups: Get "https://<LoadBalancer IP>:6443/api?timeout=10s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Addon KubeProxy is not ready: unable to retrieve kube-proxy daemonset from the guest cluster: failed to get server groups: Get "https://<LoadBalancer IP>:6443/api?timeout=10s": context deadline exceeded
  
  
• VM Operator ConfigMap shows DNS details are empty
  
  kubectl get configmaps vmoperator-network-config -n vmware-system-vmop -o yaml
apiVersion: v1
data:
  nameservers: ""  <-- DNS server details are blank
    ntpservers: <ntp server>
kind: ConfigMap
metadata:
  annotations:

Environment

vSphere Kuberenetes Service 

Cause

VM Operator manages the lifecycle of virtual machines. If the nameservers field in the vmoperator configmaps file is blank or null, the VM Operator will fail to power on the VM.

Resolution

To resolve this issue, update the Workload network DNS Configuration. Refer to Change Workload Network Settings on a Supervisor Configured with NSX

To update the DNS on Workload Network, log in to the vSphere Client and proceed with the below steps

1. In vSphere Client >  Workload Management > Supervisor > Supervisor Cluster > Configure.
2. Select  Network > Workload Network.
3. Edit DNS Server and enter the IP addresses of DNS Server and click Save.

Additional Information

To update the DNS server on a Workload Network via DCLI, use the following commands

Step 1: While connected to the VCSA as root, retrieve the cluster domain ID using the command below:

dcli com vmware vcenter namespacemanagement clusters list

Step 2:  To update the workload DNS server, use the command below. The value for --cluster should be copied from the Step 1

dcli com vmware vcenter namespacemanagement clusters update --cluster domain-c#  --worker-dns WORKER_DNS