Error "x509: certificate has expired" due to expired kubelet and kapp-controller certificates

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• Tanzu Kubernetes Grid (TKG) clusters experience control plane, metrics, or package management communication failures, with core services reporting x509: certificate has expired errors.
• The Kubernetes API server rejects communication with affected components due to TLS validation failures.
• Logs have the below errors

Subject : CN=kapp-controller@########### Issuer : CN=kapp-controller-ca@########### Not valid before : Nov 30 04:26:53 2024 GMT Not valid after : Nov 30 04:26:53 2025 GMT
The SSL certificate has already expired :

Subject : CN=###########@########### Issuer : CN=###########@###########  Not valid before : Nov 30 04:35:40 2024 GMT Not valid after : Nov 30 04:35:40 2025 GMT

•  Administrative operations such as kubectl exec, kubectl logs, and node metric scraping via metrics-server fail.
•  Package management webhooks are halted, preventing the installation, upgrade, or modification of TKG platform add-ons.
•  Tenant application availability and pod-to-pod network traffic remain unaffected as container runtimes and CNIs do not rely on these specific certificates.

 

Environment

TKG 2.5.2

Cause

• The cluster uptime has exceeded 365 days without undergoing a node rollout or lifecycle upgrade, allowing ephemeral certificates to naturally expire.
• Unlike core Kubernetes control plane certificates (which are auto-rotated by kubeadm), the Kubelet serving certificates and kapp-controller APIService utilize self-signed certificates dynamically generated during process initialization with a hardcoded 365-day validity period.
• Standard TKG lifecycle management expects clusters to undergo node rollouts at least annually, which destroys the old OS instances and automatically resets these 365-day timers.

Resolution

• To clear the expired certificates, the long-running pods must be forcefully cycled.
• The deployment controllers will immediately spawn replacements, automatically creating new self-signed certificates with 1 year validity during initialization.
• The kubelet certificates  rotation must be executed on all control plane and worker nodes.

1. Establish an SSH connection to a target node.

2. Back up and remove the expired Kubelet serving certificates from the host OS disk:
   
   sudo mv /var/lib/kubelet/pki/kubelet.crt /tmp/kubelet.crt.bak
sudo mv /var/lib/kubelet/pki/kubelet.key /tmp/kubelet.key.bak

3. Restart the Kubelet systemd service to instantly generate new 365-day certificates:
   
   sudo systemctl restart kubelet

4. Verify the new certificate was generated with an expiration date exactly one year from today:
   
   sudo openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -text -noout | grep -E 'Issuer:|Not After|Subject:'

5. Disconnect from the node and repeat steps 1-4 sequentially across every single control plane and worker node in the environment.

6. To rotate the kapp-controller certificate, authenticate to the cluster via kubectl and trigger a rolling restart of the deployment:
   
   kubectl rollout restart deployment kapp-controller -n tkg-system

7. Wait 60 seconds, then verify the APIService has registered the new certificate and shows True under the AVAILABLE column:
   
   kubectl get apiservice v1alpha1.data.packaging.carvel.dev

 

Additional Information

• VKS Guest Cluster Creation or Upgrade Hangs Due to Expired Carvel Packaging Certificate

• TKGm Certificate Rotation - Control Plane Nodes

• Apiservice shows expired certificate