Machine Fails to Bootstrap Due to Missing kubelet-config ConfigMap in VKS Cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

During a rolling update, new machines fail to bootstrap. Each new VM reaches the kubeadm phase but does not generate /etc/kubernetes/pki, /var/lib/kubelet/config.yaml, or /etc/kubernetes/admin.conf. The kubelet service does not start, and the machine status remains in a waiting state for nodeRef/Node.

In the node’s bootstrap logs located at /var/log/cloud-init-output.log and /var/log/cloud-init.log, kubeadm terminates during the kubelet-start phase with the following errors:

configmap “kubelet-config” not found

error execution phase kubelet-start: failed to load kubelet configuration: configmap “kubelet-config” not found

MachineSet events show:

Machine has unhealthy node

KubeadmControlPlane (KCP) describe output shows:

waiting for control plane/worker to pass preflight checks to continue reconciliation: Machine does not have a corresponding Node yet (Machine.status.nodeRef not set)

Reviewing /var/log/cloud-init-output.log on the affected node confirms kubeadm fails immediately after attempting to write the kubelet configuration file.

error execution phase preflight: unable to fetch kubeadm-config ConfigMap: failed to get config map:

Network connectivity and control-plane endpoint checks remain functional, isolating the issue to the missing ConfigMap.

This condition prevents new VMs from completing initialization and blocks the rolling update process.

Environment

vSphere Supervisor

VKS Cluster

Cause

The kubelet-config ConfigMap in the VKS cluster’s kube-system namespace is missing.

During bootstrap, kubeadm retrieves this ConfigMap to obtain the default kubelet configuration parameters required to create /var/lib/kubelet/config.yaml.

If the ConfigMap is absent, kubeadm cannot complete the kubelet-start phase and reports the error “configmap ‘kubelet-config’ not found,” leaving the node unregistered.

Resolution

1. From a healthy control-plane node in the same VKS cluster, collect the kubelet configuration file located at /var/lib/kubelet/config.yaml.
   
   
2. Using the contents of that file, create a new ConfigMap in the guest cluster’s kube-system namespace named kubelet-config.
   If multiple clusters exist with the same Kubernetes version, the kubelet-config ConfigMap from a working cluster can also be used as a reference to recreate the missing one.
   
   
3. Apply the ConfigMap and verify that both kubeadm-config and kubelet-config ConfigMaps are present in the kube-system namespace within the VKS cluster's context.
   
   
4. Connect into the Supervisor cluster context
   
   
5. Confirm on the VKS supervisor service version (svc-tkg.vsphere.vmware.com tkg.vsphere.vmware.com):
   

   kubectl get pkgi -n vmware-system-supervisor-services

    

6. If the VKS supervisor service version is v3.1.0 or higher, remediate the failed Machine from the Supervisor namespace so that WCP regenerates bootstrap data and creates a new control-plane VM:
   

   kubectl annotate machine -n <workload cluster namespace> <stuck provisioned machine> 'cluster.x-k8s.io/remediate-machine=""'

   On lower VKS versions, contact VMware by Broadcom Technical Support for assistance, referencing this KB.

   
   
   
7. Confirm that the new control-plane VM successfully bootstraps, generates /var/lib/kubelet/config.yaml, and registers as Ready in the VKS cluster.