Replace vSphere with Tanzu Guest Cluster/vSphere Kubernetes Cluster Certificates
Article ID: 323453
Updated On: 04-09-2025
Products
VMware vSphere ESXi
VMware vSphere with Tanzu
Issue/Introduction
Symptoms:
- vSphere Kubernetes Cluster, also known as vSphere with Tanzu Guest Cluster, certificates have expired or are about to expire.
- Use the following command while connected via SSH into either of the Guest Cluster Control Plane VMs.
-
find / -type f \( -name "*.cert" -o -name "*.crt" \) -print 2>/dev/null | egrep -iv 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd|backup' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'
-
- SSH into Guest Cluster Control Plane VM Instructions
- Main vSphere with Tanzu Cert Guide
- For vSphere with Tanzu Supervisor Cluster Certificate replacement, reference the following KB: Replace vSphere with Tanzu Supervisor Certificates
Environment
vSphere 8.0 with Tanzu
vSphere 7.0 with Tanzu
vSphere 7.0 with Tanzu
This issue can occur regardless of whether or not the cluster is managed by Tanzu Mission Control (TMC) or not.
Cause
Kubernetes certificates have a default certificate expiration time of 1 year.
VMware by Broadcom Kubernetes products adhere to this certificate expiry timeline.
Resolution
Prerequisites:
-
- Download the attached wcp_cert_manager tool from Replace vSphere with Tanzu Supervisor Certificates which can be run from either of the two locations to replace Guest Cluster certificates:
- From a jumpbox that has the kubectl and vSphere Plugin for kubectl installed that also has network connectivity to the Workload Network.
- Instructions for download and install of these utilities can be found: Download and Install the Kubernetes CLI Tools for vSphere
- From an SSH session to one of the Supervisor Control Plane nodes.
- SSH into Supervisor Control Plane VM Instructions: Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's
- From a jumpbox that has the kubectl and vSphere Plugin for kubectl installed that also has network connectivity to the Workload Network.
- The Kubernetes API and ETCD servers are healthy in the guest cluster
- The vmware-system-user is not expired. If vmware-system-user is expired, see vmware-system-user account expired KB
- Download the attached wcp_cert_manager tool from Replace vSphere with Tanzu Supervisor Certificates which can be run from either of the two locations to replace Guest Cluster certificates:
Installation:
- Move the attached file titled wcp_cert_manager.zip to the jumpbox where vSphere plugin for kubectl is installed or Supervisor CP node. (Use WinSCP from Windows OS's if required):
-
scp ./wcp_cert_manager.zip root@<SUPERVISOR_VM_IP>:/root
-
Example Output:
The authenticity of host '<SUPERVISOR_VM_IP> (<SUPERVISOR_VM_IP>)' can't be established.
ECDSA key fingerprint is SHA256:<SUPERVISOR_VM_ECDSA_FINGERPRINT>.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '<SUPERVISOR_VM_IP>' (ECDSA) to the list of known hosts.
VMware vCenter Server 7.0.3.01000
Type: vCenter Server with an embedded Platform Services Controller
Password:
wcp_cert_manager.zip 100% 8473KB 8.3MB/s 00:00
- Unzip the file and move it to the executable path:
# unzip wcp_cert_manager.zip
Archive: wcp_cert_manager.zip
inflating: certmgr
# ls -l
total 30956
-rwxr-xr-x 1 root root 23019418 Nov 28 01:24 certmgr
-rw-r--r-- 1 root root 8675846 Jan 17 16:09 wcp_cert_manager.zip
# cp certmgr /usr/bin/
Execution:
- List Guest Cluster certificates:
certmgr tkc certificates list -n <NAMESPACE_NAME> <CLUSTER_NAME>
Example Output:
# certmgr tkc certificates list -n certs cluster1
20:53:04 proc.go:267: [/root/certmgr tkc certificates list -n certs cluster1]
20:53:04 list.go:20: checking certs on machine, kind: Machine, namespace: certs, name: <CONTROL_PLANE_VM_NAME>, ip: <CONTROL_PLANE_VM_IP>
20:53:04 client.go:196: copying certmgr to remote, kind: Machine, namespace: certs, name: <CONTROL_PLANE_VM_NAME>
20:53:05 scp.go:86: copying file certmgr to /home/vmware-system-user/certmgr with size 64 MiB, mode 750
Uploading 64 MiB/64 MiB
20:53:06 client.go:196: finished copying
/etc/bash.bashrc: line 43: TMOUT: readonly variable
/etc/bash.bashrc: line 43: TMOUT: readonly variable
20:53:06 proc.go:267: [/root/certmgr certificates list]
20:53:06 proc.go:267: program exited
+--------------+-----------------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
| SCOPE | IP | HOSTNAME | NAME | NOTAFTER | ISEXPIRED |
+--------------+-----------------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
| controlplane | <CONTROL_PLANE_VM_IP> | <CONTROL_PLANE_VM_NAME> | /etc/kubernetes/pki/front-proxy-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/apiserver.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/apiserver-etcd-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/apiserver-kubelet-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet.crt | 2024-10-26 15:11:59 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet-client-current.pem | 2024-10-26 15:22:47 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/server.crt | 2024-09-17 15:02:58 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/peer.crt | 2024-10-26 15:32:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/healthcheck-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/front-proxy-ca.crt | 2033-09-15 14:57:11 +0000 UTC | false |
| | | | /etc/kubernetes/pki/ca.crt | 2033-09-15 14:57:11 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/ca.crt | 2033-09-15 14:57:12 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet.crt | 2024-10-26 15:11:59 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet-client-current.pem | 2024-10-26 15:22:47 +0000 UTC | false |
+--------------+-----------------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
20:53:06 list.go:54: command execution completed successfully.
20:53:06 proc.go:267: program exited
- Rotate vSphere Kubernetes Cluster/Guest Cluster certificates:
certmgr tkc certificates rotate -n <NAMESPACE_NAME> <CLUSTER_NAME>
Example Output:
certmgr tkc certificates rotate -n certs cluster1
Uploading 64 MiB/64 MiB
/etc/bash.bashrc: line 43: TMOUT: readonly variable
/etc/bash.bashrc: line 43: TMOUT: readonly variable
20:58:17 proc.go:267: [/root/certmgr certificates rotate]
20:58:22 etcd_actions.go:66: etcd healthy after 0.83 seconds
20:58:22 root.go:265: result {[{backup certificates /root } {rotate etcd server certificate true } {rotate api server etcd client certificate true } {rotate etcd peer certificate true } {rotate etcd health check certificate true } {rotate api server certificate true } {rotate kubelet client api server certificate true } {rotate front proxy certificate true } {rotate controller-manager certificate true } {rotate scheduler certificate true } {rotate kubelet certificate <nil> } {rotate kubeadm admin certificate true } {verify etcd health true }] ok <nil>}
20:58:22 proc.go:267: program exited
+-----------------------------------------------------+----------------+
| TASKS | OVERALL STATUS |
+-----------------------------------------------------+----------------+
| +--------------------------------+--------+-------+ | ok |
| | TASK | RESULT | ERROR | | |
| +--------------------------------+--------+-------+ | |
| | backup certificates | /root | | | |
| | rotate etcd server certificate | true | | | |
| | rotate api server etcd client | true | | | |
| | certificate | | | | |
| | rotate etcd peer certificate | true | | | |
| | rotate etcd health check | true | | | |
| | certificate | | | | |
| | rotate api server certificate | true | | | |
| | rotate kubelet client api | true | | | |
| | server certificate | | | | |
| | rotate front proxy certificate | true | | | |
| | rotate controller-manager | true | | | |
| | certificate | | | | |
| | rotate scheduler certificate | true | | | |
| | rotate kubelet certificate | | | | |
| | rotate kubeadm admin | true | | | |
| | certificate | | | | |
| | verify etcd health | true | | | |
| +--------------------------------+--------+-------+ | |
| | |
+-----------------------------------------------------+----------------+
- Restart kube-controller-manager pods
- On each control plane node, run the following:
-
crictl rm -f $(crictl ps --label io.kubernetes.container.name=kube-controller-manager -q)
-
crictl rm -f $(crictl ps --label io.kubernetes.container.name=kube-scheduler -q)
-
NOTE:
- This tool replaces only the ControlPlane certificates in Guest Clusters. The certs on the worker nodes can expire with no impact to functionality to the cluster so they do not need to be rotated.
- All logs for this tool are logged under /var/log/vmware/certmgr.log
Additional Information
Main vSphere with Tanzu Cert Page: https://knowledge.broadcom.com/external/article?articleNumber=323421
Feedback
Was this article helpful?
Yes
No
Powered by
