Guest Cluster Control Plane Node Upgrade stuck: Nameserver Entry Missing in ConfigMap "No nameservers in vmoperator-network-config ConfigMap"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

A TKC upgrade was initiated by the user. During the process, one control plane node failed to receive an IP address. VM logs indicated missing nameservers in the vmoperator-network-config ConfigMap, resulting in DNS resolution failure and blocking the upgrade.

Vmop pod log shows "No nameservers in vmoperator-network-config" error pointing out to nameservers not being added

kubectl logs -n <vmop-namespace> <vmop-controller-pod> | grep <control-plane-vm-name>


E0808 08:44:59.548520       1 controller.go:329] "Reconciler error" err="No nameservers in vmoperator-network-config ConfigMap" controller="virtualmachine" controllerGroup="vmoperator.vmware.com" controllerKind="VirtualMachine" VirtualMachine="<namespace>/<vm-name>" namespace="<namespace>" name="<vm-name>" reconcileID="<masked-id>"
I0808 08:45:00.667470       1 virtualmachineservice_controller.go:628] "Skipping VM without primary IP assigned" logger="VirtualMachineService" name="<namespace>/<vm-service-name>" virtualMachine="<namespace>/<vm-name>"
I0808 08:45:20.029127       1 virtualmachine_controller.go:309] "Reconciling VirtualMachine" logger="VirtualMachine" name="<namespace>/<vm-name>"
I0808 08:45:20.084267       1 virtualmachine_controller.go:318] "Finished Reconciling VirtualMachine" logger="VirtualMachine" name="<namespace>/<vm-name>"
E0808 08:45:20.084287       1 virtualmachine_controller.go:243] "Failed to reconcile VirtualMachine" err="No nameservers in vmoperator-network-config ConfigMap" logger="VirtualMachine" name="<namespace>/<vm-name>"
E0808 08:45:20.084862       1 controller.go:329] "Reconciler error" err="No nameservers in vmoperator-network-config ConfigMap" controller="virtualmachine" controllerGroup="vmoperator.vmware.com" controllerKind="VirtualMachine" VirtualMachine="<namespace>/<vm-name>" namespace="<namespace>" name="<vm-name>" reconcileID="<masked-id>"
I0808 08:46:01.045651       1 virtualmachine_controller.go:309] "Reconciling VirtualMachine" logger="VirtualMachine" name="<namespace>/<vm-name>"
I0808 08:46:01.073752       1 virtualmachine_controller.go:340] "Finished Reconciling VirtualMachine" logger="VirtualMachine" name="<namespace>/<vm-name>"

Environment

VMware vSphere Kubernetes Service

Cause

VM Operator (vmop) pod logs revealed repeated failures in VM provisioning. No DNS resolution was occurring during node creation.

Resolution

1. Edit the ConfigMap Update the vmoperator-network-config ConfigMap to include the required nameservers as per user specifications
   
   kubectl edit configmap -n <vmop-namespace> vmoperator-network-config -o yaml
apiVersion: v1
data:
  nameservers: <masked-ip-1> <masked-ip-2>
  ntpservers: <masked-ntp-1> <masked-ntp-2>
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
    ...
   
   
2. Roll Out VM Operator Pods Restart the vmware-system-vmop-controller-manager pods to apply the updated network configuration
   
   kubectl rollout restart deployment -n <vmop-namespace> <vmop-controller-deployment>
deployment.apps/<vmop-controller-deployment> restarted
   
   
3. Upgrade Trigger Once the pods are rolled out, the control plane VM will reattempt provisioning. If DNS resolution succeeds, the TKC upgrade will automatically resume and progress.