Control plane node certificates expired after failed automatic renewal in Telco Cloud Automation

Products

• VMware Telco Cloud Automation

Issue/Introduction

• Control plane nodes are offline or failing.

• The output falls back to invalid dates and displays the error message: !MISSING! super-admin.conf.

• Running the command kubeadm certs check-expiration on a control plane node throws an error and displays missing configuration files (e.g., admin.conf, apiserver, controller-manager.conf).

Environment

TCA: 2.3, 3.4

Cause

This issue occurs when the standard Kubernetes certificates on the control plane node either expire or lose their context/sync.

While TCA 2.3 and later feature automatic certificate renewal for TKGm clusters, this auto-renewal workflow requires the cluster to be completely healthy. If any node is stuck in a provisioning state, or if the etcd quorum is not intact, the automatic workflow will halt. If the cluster falls into an expired state before the automated workflow can succeed, a manual certificate renewal is required

Resolution

To resolve the missing certificates and bring the control plane node back online, perform a manual certificate rotation on all three Control Plane nodes sequentially.

1. Access the Control Plane Node: SSH into the affected Control Plane Node and elevate to root privileges:

   Bash

   ssh capv@<K8S-CONTROL-PLANE-IP>
   sudo -i
   

2. Execute the Renewal Command: Run the following command to force the renewal of all core Kubernetes certificates:

   Bash

   kubeadm certs renew all
   

3. Restart the Core Pods: The static pods must be restarted to pick up the newly generated certificates. Generate and run the stop commands for kube-apiserver, kube-controller-manager, kube-scheduler, and etcd:

   Bash

   crictl stop $(crictl ps --name kube-apiserver -q)
   crictl stop $(crictl ps --name kube-controller-manager -q)
   crictl stop $(crictl ps --name kube-scheduler -q)
   crictl stop $(crictl ps --name etcd -q)
   

   Note: If your environment uses kube-vip, you must also run:

   Bash

   crictl stop $(crictl ps --name kube-vip -q)
   

4. Verify Rotation: Check the certificate expiration again to ensure they are now valid and the !MISSING! error has cleared:

   Bash

   kubeadm certs check-expiration
   

5. Repeat on Remaining Nodes: You MUST repeat Steps 1 through 4 sequentially on all 3 Control Plane nodes.

6. Re-sync the Kubeconfig: Once the certificates are rotated on all nodes, the cluster API will come back online. However, because the cluster certificates have changed, the kubeconfig stored inside the TCA Database is now invalid.

   You must re-sync the Kubeconfig in TCA-M and TCA-CP by either:

   • Leveraging the one-click rotation workflow introduced in the TCA 3.4 UI.

   • Using the hybridity APIs.

Additional Information

Manually renew cluster certificates