Node rollout fails immediately after adding a custom label to a node pool

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

After adding a custom label to the node-pool section in the cluster YAML, a new node spins up, but the rollout process hangs or fails to complete.

Checking the kubelet logs on the affected node reveals the following error:

# journalctl -u kubelet
failed to validate kubelet flags: unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: [XXXX.kubernetes.io/component]

Environment

vSphere Kubernetes Service

Cause

This issue occurs because the custom label uses the restricted kubernetes.io (or k8s.io) namespace.

For security reasons, the Kubernetes NodeRestriction admission controller prevents kubelets from self-applying or modifying labels with the *.kubernetes.io or *.k8s.io prefixes. This prevents compromised nodes from altering their core topology labels or escalating privileges.

For more details, refer to the Kubernetes Admission Controllers Reference.

Resolution

To resolve this issue, change the label key to use a non-reserved namespace.