Update DNS on vSphere Kubernetes Service VKS Cluster after changing DNS servers

Products

• VMware vSphere Kubernetes Service
• Tanzu Kubernetes Runtime

Issue/Introduction

This KB is intended to provide a workaround to push DNS changes to vSphere Kubernetes Service (VKS) Clusters effectively.

Symptoms:  

After updating DNS servers for the Supervisor Cluster's network configuration through the vSphere Web Client -> Supervisor Cluster -> Configure -> Network -> 'Management Network' and 'Workload Network', running pods and new pods on VKS Clusters do not recognize the changes.

Environment

vSphere Supervisor

 

VKS Cluster

Cause

This behavior occurs on VKS Clusters and not on Supervisor Cluster because the Supervisor Cluster's CoreDNS pod forwards requests directly to the configured DNS and updates immediately when changes are made.

 

However, the VKS Cluster's CoreDNS pod forwards requests to the VKS Cluster node DNS configuration, which then relies on the node's resolv.conf to point to the DNS configured. The network file on the Workload Cluster node, located at /etc/systemd/network/10-gosc-eth0.network, only updates when nodes are recreated.

Resolution

There is no resolution for this issue, as the functionality operates as intended by design.

Workaround:

To work around this issue, you can recreate the nodes by initiating a rolling redeployment of the VKS cluster's nodes.

For example, this can be done by changing the vmClass in the cluster's YAML, which triggers a rolling update and updates the host file on the nodes upon recreation.

 

If changing the cluster's configuration is not feasible (e.g., due to policy constraints or resource compatibility), please see the below alternative:

On ClusterClass v3.3.X and higher, a label can be applied to the VKS cluster's cluster YAML to initiate a rolling redeployment:

See: How to Force a Cluster Configuration Update/Manual Rolling Redeployment of Nodes in a vSphere Kubernetes Cluster on ClusterClass v3.3.X

Additional Information

Impact/Risks:

DNS lookups performed by workloads running in VKS Clusters may fail due to this issue.