TKC Control plane nodes were unexpectedly recreated in VKS(TKGs)

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

TKC all control plane nodes were suddenly recreated, while the worker nodes remained unaffected.

# kubectl get nodes
NAME                                                         STATUS   ROLES           AGE   VERSION
<CLUSTER NAME>-tkg-cluster-nodepool-nn8w-swsd9-txvfc-75h6v   Ready    <none>         330d   v1.29.4+vmware.3-fips.1
<CLUSTER NAME>-zcptp-5vm79                                   Ready    control-plane   10m   v1.29.4+vmware.3-fips.1

Environment

VMware vSphere Kubernetes Service - vCenter 8U3 or later

Cause

This is the automatic certificate renewal feature for VKS Cluster implemented in vSphere Supervisor 8.0 U3.
The control plane node is rolled out for the certificate renewal.

kubectl vsphere login --insecure-skip-tls-verify --server <Supervisor Endpoint>
kubectl config use-context <Supervisor context>
kubectl logs -n svc-tkg-domain-<your cluster domain id> deployments/capi-kubeadm-control-plane-controller-manager

1 controller.go:473] "Rolling out Control Plane machines: Machine <Cluster Name>-xxxxx-7v5xh needs rollout: certificates will expire soon, rolloutBefore expired,Machine <Cluster Name>-xxxxx-rv2x4 needs rollout: certificates will expire soon, rolloutBefore expired,Machine <Cluster Name>-xxxxx-mdmz9 needs rollout: certificates will expire soon, rolloutBefore expired" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" KubeadmControlPlane="<Namespace>/<Cluster Name>-xxxxx" namespace="<Namespace>" name="<Cluster Name>-xxxxx" reconcileID="a928ff0b-xxxx-xxxx-xxxx-9ce4a5438779" Cluster="<Namespace>/<Cluster Name>" machinesNeedingRollout=["<Cluster Name>-xxxxx-7v5xh","<Cluster Name>-xxxxx-rv2x4","<Cluster Name>-xxxxx-mdmz9"]

Resolution

To change behavior of automatic certificate renewal, you can edit the cluster as below:

kubectl vsphere login --insecure-skip-tls-verify --server <Supervisor Endpoint>
kubectl config use-context <Supervisor context>
kubectl edit -n <Cluster namespace> cluster <Cluster name>

// under .spec.topology.variables
...
   - name: controlPlaneCertificateRotation
     value:
       activate: true # <--- Change to "false" if you want to disable it
       daysBefore: 90 # <--- Control the date of the auto renewal

Additional Information

A similar rollout occurs in a Tanzu Kubernetes Grid v2.5.x environment.
Kubernetes Control Plane nodes get recreated unexpectedly