Kubernetes Control Plane nodes get recreated unexpectedly
search cancel

Kubernetes Control Plane nodes get recreated unexpectedly

book

Article ID: 402121

calendar_today

Updated On:

Products

Tanzu Kubernetes Runtime

Issue/Introduction

All kubernetes control plane nodes were suddenly recreated, wh ile the worker nodes remained unaffected.

kubectl get nodes

NAME                                      STATUS   ROLES           AGE
<CLUSTER_NAME>-controlplane-zcq5f-xxxxx   Ready    control-plane   10m     # <--- control-plane node 1 was recreated
<CLUSTER_NAME>-controlplane-zcq5f-yyyyy   Ready    control-plane   15m     # <--- control-plane node 2 was recreated
<CLUSTER_NAME>-controlplane-zcq5f-zzzzz   Ready    control-plane   20m     # <--- control-plane node 3 was recreated
<CLUSTER_NAME>-md-0-4wjc5-524x5-7f8kl     Ready    <none>          330d
<CLUSTER_NAME>-md-0-4wjc5-524x5-fhvqh     Ready    <none>          330d
<CLUSTER_NAME>-md-0-4wjc5-524x5-hhrxs     Ready    <none>          330d

Environment

Tanzu Kuberntes Grid v2.5.x

Cause

Control Plane Node Certificate Auto-Renewal function was triggered.

kubectl config use-context <MANAGEMENT_CLUSTER_CONTEXT>
kubectl -n capi-kubeadm-control-plane-system logs deployment/capi-kubeadm-control-plane-controller-manager | grep 'will expire'

I0706 hh:mm:ss.ssss       1 controller.go:409] "Rolling out Control Plane machines: Machine <CLUSTER_NAME>-controlplane-#####-xxxxx needs rollout: certificates will expire soon, rolloutBefore expired" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" KubeadmControlPlane="default/<CLUSTER_NAME>-controlplane-#####" namespace="default" name="<CLUSTER_NAME>-controlplane-#####" reconcileID="****-****-****-****-*************" Cluster="<CLUSTER_NAMESPACE>/<CLUSTER_NAME>" machinesNeedingRollout=["<CLUSTER_NAME>-controlplane-#####-xxxxx"]

Resolution

This control plane node auto-rolling update has no negative impact. However, if you want to change behaviour of automatic certificate renewal, you can edit the cluster as below:

kubectl config use-context <MANAGEMENT_CLUSTER_CONTEXT>
kubectl edit cluster <TARGET_CLUSTER_NAME>

    - name: controlPlaneCertificateRotation
      value:
        activate: true # <--- Change to "false" if you want to disable it
        daysBefore: 90 # <--- Control the date of the auto renewal

 

If you want to know when next certificate auto-renewal will occur, you may check the certificate's expiration date like below:

Example: If daysBefore is set to 90 days, the next certificate auto-renewal process will be triggered on 2026-03-01.

kubectl get ma <TARGET_CLUSTER_CONTROLPLANE_NODE> -oyaml | grep certicatesExpiryDate
#>   certificatesExpiryDate: "2026-06-01T06:44:49Z"

Additional Information

Powered by Wolken Software