Unable to assign or unable to propogate node level label changes

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• When adding node-level labels in TKG 2.x (Cluster API v1beta1), editing metadata.labels under spec.topology.workers.machineDeployments[] only attaches labels to management cluster resources (MachineDeployment and Machine CRDs). It does not propagate to the underlying Kubernetes Node objects.

Environment

TKGm 2.5.X

Resolution

To correctly assign node labels, uUtargeted worker pool, provisioning new nodes with the label attached at boot time without workload downtime.

Pre-Checks

• Identify the index of the target MachineDeployment: kubectl get cluster <CLUSTER_NAME> -n <NAMESPACE> -o jsonpath='{range .spec.topology.workers.machineDeployments[*]}{.name}{"\n"}{end}'
  Note the 0-based index of your target worker pool
• Verify existing variables on the target MachineDeployment:
  kubectl get cluster <CLUSTER_NAME> -n <NAMESPACE> -o yaml | grep -B5 -A15 "name: <TARGET_MD_NAME>"
  If a variables: block already exists under the pool definition, adjust the patch to append to variables.overrides rather than creating a new block.

Apply the Topology Variable Patch

Replace <INDEX>, <KEY>, and <VALUE> with your environment parameters:

kubectl patch cluster <CLUSTER_NAME> -n <NAMESPACE> --type='json' -p='[
 {
  "op": "add",
  "path": "/spec/topology/workers/machineDeployments/<INDEX>/variables",
  "value": {
   "overrides": [
    {
     "name": "nodePoolLabels",
     "value": [
       {"key": "<KEY>", "value": "<VALUE>"}
     ]
    }
   ]
  }
 }
]'

Monitor Rolling Update

Cluster API will generate a new KubeadmConfigTemplate and automatically trigger a rolling replacement of the worker nodes. Monitor the progress on the Management Cluster:
kubectl get machines -n <NAMESPACE> -l topology.cluster.x-k8s.io/deployment-name=<TARGET_MD_NAME> -w

Verification

• Verify KubeadmConfigTemplate bootstrap arguments:

          NEW_KCT=$(kubectl get machinedeployment -n <NAMESPACE> -l topology.cluster.x-k8s.io/deployment-name=<TARGET_MD_NAME> -o jsonpath='{.items[0].spec.template.spec.bootstrap.configRef.name}')

         kubectl get kubeadmconfigtemplate $NEW_KCT -n <NAMESPACE> -o yaml | grep -A5 kubeletExtraArgs

• Verify Node labels inside the Workload Cluster:
   Switch to the Workload Cluster context: kubectl get nodes -L <KEY>