Replace vSphere with Tanzu Guest Cluster Certificates
Article ID: 323453
Updated On:
Products
VMware vSphere ESXi
VMware vSphere with Tanzu
Issue/Introduction
Symptoms:
- vSphere with Tanzu Guest Cluster certificates have expired or are about to expire.
- Use the following command while connected via SSH into either of the Guest Cluster Control Plane VMs.
# find / -type f \( -name "*.cert" -o -name "*.crt" \) -print 2>/dev/null | egrep -iv 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd|backup' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'
- SSH into Guest Cluster Control Plane VM Instructions
- Main vSphere with Tanzu Cert Guide
- For vSphere with Tanzu Supervisor Cluster Certificate replacement, reference the following KB: Replace vSphere with Tanzu Supervisor Certificates
Environment
VMware vSphere 8.0 with Tanzu
VMware vSphere 7.0 with Tanzu
VMware vSphere 7.0 with Tanzu
Resolution
Prerequisites:
-
- Download the attached wcp_cert_manager tool from this kb which can be run from either of the two locations to replace Guest Cluster certificates:
- From a jumpbox that has the kubectl and vSphere Plugin for kubectl installed that also has network connectivity to the Workload Network. Instructions for download and install of these utilities can be found here
- From an SSH session to one of the Supervisor ControlPlane nodes. SSH into Supervisor Control Plane VM Instructions: Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's (90194)
- The Kubernetes API and ETCD servers are healthy
- The vmware-system-user is not expired
- Download the attached wcp_cert_manager tool from this kb which can be run from either of the two locations to replace Guest Cluster certificates:
Installation:
- Move the attached file titled wcp_cert_manager.zip to the jumpbox where vSphere plugin for kubectl is installed or Supervisor CP node. (Use WinSCP from Windows OS's if required):
# scp ./wcp_cert_manager.zip root@<SUPERVISOR_VM_IP>:/root
Example Output:
The authenticity of host '<SUPERVISOR_VM_IP> (<SUPERVISOR_VM_IP>)' can't be established.
ECDSA key fingerprint is SHA256:<SUPERVISOR_VM_ECDSA_FINGERPRINT>.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '<SUPERVISOR_VM_IP>' (ECDSA) to the list of known hosts.
VMware vCenter Server 7.0.3.01000
Type: vCenter Server with an embedded Platform Services Controller
Password:
wcp_cert_manager.zip 100% 8473KB 8.3MB/s 00:00
ECDSA key fingerprint is SHA256:<SUPERVISOR_VM_ECDSA_FINGERPRINT>.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '<SUPERVISOR_VM_IP>' (ECDSA) to the list of known hosts.
VMware vCenter Server 7.0.3.01000
Type: vCenter Server with an embedded Platform Services Controller
Password:
wcp_cert_manager.zip 100% 8473KB 8.3MB/s 00:00
- Unzip the file and move it to the executable path:
# unzip wcp_cert_manager.zip
Archive: wcp_cert_manager.zip
inflating: certmgr
# ls -l
total 30956
-rwxr-xr-x 1 root root 23019418 Nov 28 01:24 certmgr
-rw-r--r-- 1 root root 8675846 Jan 17 16:09 wcp_cert_manager.zip
# cp certmgr /usr/bin/
Archive: wcp_cert_manager.zip
inflating: certmgr
# ls -l
total 30956
-rwxr-xr-x 1 root root 23019418 Nov 28 01:24 certmgr
-rw-r--r-- 1 root root 8675846 Jan 17 16:09 wcp_cert_manager.zip
# cp certmgr /usr/bin/
Execution:
- List Guest Cluster certificates:
# certmgr tkc certificates list -n <NAMESPACE_NAME> <CLUSTER_NAME>
Example Output:
# certmgr tkc certificates list -n certs cluster1
20:53:04 proc.go:267: [/root/certmgr tkc certificates list -n certs cluster1]
20:53:04 list.go:20: checking certs on machine, kind: Machine, namespace: certs, name: <CONTROL_PLANE_VM_NAME>, ip: <CONTROL_PLANE_VM_IP>
20:53:04 client.go:196: copying certmgr to remote, kind: Machine, namespace: certs, name: <CONTROL_PLANE_VM_NAME>
20:53:05 scp.go:86: copying file certmgr to /home/vmware-system-user/certmgr with size 64 MiB, mode 750
Uploading 64 MiB/64 MiB
20:53:06 client.go:196: finished copying
/etc/bash.bashrc: line 43: TMOUT: readonly variable
/etc/bash.bashrc: line 43: TMOUT: readonly variable
20:53:06 proc.go:267: [/root/certmgr certificates list]
20:53:06 proc.go:267: program exited
+--------------+-----------------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
| SCOPE | IP | HOSTNAME | NAME | NOTAFTER | ISEXPIRED |
+--------------+-----------------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
| controlplane | <CONTROL_PLANE_VM_IP> | <CONTROL_PLANE_VM_NAME> | /etc/kubernetes/pki/front-proxy-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/apiserver.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/apiserver-etcd-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/apiserver-kubelet-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet.crt | 2024-10-26 15:11:59 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet-client-current.pem | 2024-10-26 15:22:47 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/server.crt | 2024-09-17 15:02:58 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/peer.crt | 2024-10-26 15:32:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/healthcheck-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/front-proxy-ca.crt | 2033-09-15 14:57:11 +0000 UTC | false |
| | | | /etc/kubernetes/pki/ca.crt | 2033-09-15 14:57:11 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/ca.crt | 2033-09-15 14:57:12 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet.crt | 2024-10-26 15:11:59 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet-client-current.pem | 2024-10-26 15:22:47 +0000 UTC | false |
+--------------+-----------------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
20:53:06 list.go:54: command execution completed successfully.
20:53:06 proc.go:267: program exited
20:53:04 proc.go:267: [/root/certmgr tkc certificates list -n certs cluster1]
20:53:04 list.go:20: checking certs on machine, kind: Machine, namespace: certs, name: <CONTROL_PLANE_VM_NAME>, ip: <CONTROL_PLANE_VM_IP>
20:53:04 client.go:196: copying certmgr to remote, kind: Machine, namespace: certs, name: <CONTROL_PLANE_VM_NAME>
20:53:05 scp.go:86: copying file certmgr to /home/vmware-system-user/certmgr with size 64 MiB, mode 750
Uploading 64 MiB/64 MiB
20:53:06 client.go:196: finished copying
/etc/bash.bashrc: line 43: TMOUT: readonly variable
/etc/bash.bashrc: line 43: TMOUT: readonly variable
20:53:06 proc.go:267: [/root/certmgr certificates list]
20:53:06 proc.go:267: program exited
+--------------+-----------------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
| SCOPE | IP | HOSTNAME | NAME | NOTAFTER | ISEXPIRED |
+--------------+-----------------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
| controlplane | <CONTROL_PLANE_VM_IP> | <CONTROL_PLANE_VM_NAME> | /etc/kubernetes/pki/front-proxy-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/apiserver.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/apiserver-etcd-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/apiserver-kubelet-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet.crt | 2024-10-26 15:11:59 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet-client-current.pem | 2024-10-26 15:22:47 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/server.crt | 2024-09-17 15:02:58 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/peer.crt | 2024-10-26 15:32:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/healthcheck-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/front-proxy-ca.crt | 2033-09-15 14:57:11 +0000 UTC | false |
| | | | /etc/kubernetes/pki/ca.crt | 2033-09-15 14:57:11 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/ca.crt | 2033-09-15 14:57:12 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet.crt | 2024-10-26 15:11:59 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet-client-current.pem | 2024-10-26 15:22:47 +0000 UTC | false |
+--------------+-----------------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
20:53:06 list.go:54: command execution completed successfully.
20:53:06 proc.go:267: program exited
- Rotate Guest Cluster certificates:
# certmgr tkc certificates rotate -n <NAMESPACE_NAME> <CLUSTER_NAME>
Example Output:
# certmgr tkc certificates rotate -n certs cluster1
Uploading 64 MiB/64 MiB
/etc/bash.bashrc: line 43: TMOUT: readonly variable
/etc/bash.bashrc: line 43: TMOUT: readonly variable
20:58:17 proc.go:267: [/root/certmgr certificates rotate]
20:58:22 etcd_actions.go:66: etcd healthy after 0.83 seconds
20:58:22 root.go:265: result {[{backup certificates /root } {rotate etcd server certificate true } {rotate api server etcd client certificate true } {rotate etcd peer certificate true } {rotate etcd health check certificate true } {rotate api server certificate true } {rotate kubelet client api server certificate true } {rotate front proxy certificate true } {rotate controller-manager certificate true } {rotate scheduler certificate true } {rotate kubelet certificate <nil> } {rotate kubeadm admin certificate true } {verify etcd health true }] ok <nil>}
20:58:22 proc.go:267: program exited
+-----------------------------------------------------+----------------+
| TASKS | OVERALL STATUS |
+-----------------------------------------------------+----------------+
| +--------------------------------+--------+-------+ | ok |
| | TASK | RESULT | ERROR | | |
| +--------------------------------+--------+-------+ | |
| | backup certificates | /root | | | |
| | rotate etcd server certificate | true | | | |
| | rotate api server etcd client | true | | | |
| | certificate | | | | |
| | rotate etcd peer certificate | true | | | |
| | rotate etcd health check | true | | | |
| | certificate | | | | |
| | rotate api server certificate | true | | | |
| | rotate kubelet client api | true | | | |
| | server certificate | | | | |
| | rotate front proxy certificate | true | | | |
| | rotate controller-manager | true | | | |
| | certificate | | | | |
| | rotate scheduler certificate | true | | | |
| | rotate kubelet certificate | | | | |
| | rotate kubeadm admin | true | | | |
| | certificate | | | | |
| | verify etcd health | true | | | |
| +--------------------------------+--------+-------+ | |
| | |
+-----------------------------------------------------+----------------+
Uploading 64 MiB/64 MiB
/etc/bash.bashrc: line 43: TMOUT: readonly variable
/etc/bash.bashrc: line 43: TMOUT: readonly variable
20:58:17 proc.go:267: [/root/certmgr certificates rotate]
20:58:22 etcd_actions.go:66: etcd healthy after 0.83 seconds
20:58:22 root.go:265: result {[{backup certificates /root } {rotate etcd server certificate true } {rotate api server etcd client certificate true } {rotate etcd peer certificate true } {rotate etcd health check certificate true } {rotate api server certificate true } {rotate kubelet client api server certificate true } {rotate front proxy certificate true } {rotate controller-manager certificate true } {rotate scheduler certificate true } {rotate kubelet certificate <nil> } {rotate kubeadm admin certificate true } {verify etcd health true }] ok <nil>}
20:58:22 proc.go:267: program exited
+-----------------------------------------------------+----------------+
| TASKS | OVERALL STATUS |
+-----------------------------------------------------+----------------+
| +--------------------------------+--------+-------+ | ok |
| | TASK | RESULT | ERROR | | |
| +--------------------------------+--------+-------+ | |
| | backup certificates | /root | | | |
| | rotate etcd server certificate | true | | | |
| | rotate api server etcd client | true | | | |
| | certificate | | | | |
| | rotate etcd peer certificate | true | | | |
| | rotate etcd health check | true | | | |
| | certificate | | | | |
| | rotate api server certificate | true | | | |
| | rotate kubelet client api | true | | | |
| | server certificate | | | | |
| | rotate front proxy certificate | true | | | |
| | rotate controller-manager | true | | | |
| | certificate | | | | |
| | rotate scheduler certificate | true | | | |
| | rotate kubelet certificate | | | | |
| | rotate kubeadm admin | true | | | |
| | certificate | | | | |
| | verify etcd health | true | | | |
| +--------------------------------+--------+-------+ | |
| | |
+-----------------------------------------------------+----------------+
- Restart kube-controller-manager pods
- On each control plane node, run the following:
- crictl rm -f $(crictl ps --label io.kubernetes.container.name=kube-controller-manager -q)
- crictl rm -f $(crictl ps --label io.kubernetes.container.name=kube-scheduler -q)
NOTE:
- This tool replaces ONLY the ControlPlane certificates in Guest Clusters currently. Worker node cert replacement must be applied with a worker node rollout.
- All logs for this tool are logged under /var/log/vmware/certmgr.log
Additional Information
Main vSphere with Tanzu Cert Page: https://knowledge.broadcom.com/external/article?legacyId=89324
Feedback
Yes
No
Powered by
