VKS Guest Cluster Deployment Hangs Due to DNS Resolution Failures

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

VKS guest cluster deployment stalls during the control plane VM initialization phase. Although virtual machines are created, they fail to execute startup tasks.

• vmop Logs: The vmop logs on the Supervisor Control VM report a reconciliation failure:

"Failed to reconcile VirtualMachine" err="persistent volume: #### not attached to VM"

"Reconciler error" err="persistent volume: #### not attached to VM" controller="virtualmachine"

• CSI Logs: The CSI logs indicate a failure to determine the CNS volume type due to DNS resolution errors:

error syncing volume "pvc-####": rpc error: code = Internal desc = failed to determine CNS volume type for volume: "######". Error: rpc error: code = Internal desc = queryVolume failed for volumeID: "#####" with error= Post "https://<vCenter_fqdn>:443/sdk": dial tcp: lookup <vCenter_fqdn> on 127.0.0.53:53: no such host

rpc error: code = Internal desc = failed to provision volume with StorageClass "vsan-default-storage-policy": rpc error: code = Internal desc = failed to get vCenter from Manager. Error: Post "<vCenter_fqdn>:443/sdk": dial tcp: lookup <vCenter_fqdn> on 127.0.0.53:53: no such host

• Supervisor Status: The Supervisor configuration reports a network identifier resolution error:

Configured Supervisor Control plan VM's Management Network Configuration error

Unable to resolve the vCenter Primary Network Identifier <vCenter_fqdn> with the configured management DNS server(s) on control plane VM.

Validate that the management DNS servers 'IP-1, IP-2, IP-3' can resolve <vCenter_fqdn>

Environment

VMware vSphere Kubernetes Service

Cause

Invalid or unreachable DNS server configurations in the Supervisor Cluster management settings prevent the resolution of critical infrastructure FQDNs (e.g., the vCenter Server FQDN). When the Supervisor cannot resolve the vCenter identifier, the CSI driver fails to attach persistent volumes to the control plane VM, preventing the VM from completing its boot process.

Resolution

1. Log in to the vCenter Server and navigate to the Workload Management > Supervisors, choose the supervisor. Navigate to Configure > Network.
2. Review the DNS server configuration in the Management Network section.
3. Remove any unresponsive or incorrect DNS server IPs.
4. Ensure the configured DNS servers can resolve the vCenter FQDN.
5. Validate using nslookup <vcenter-fqdn> from the Supervisor Control Plane VM (via SSH).
6. Once DNS is verified, trigger a reconciliation or redeploy the impacted guest cluster.