Prerequisites:
The attached wcp_cert_manager tool can be run from either of two locations to replace Guest Cluster certificates:
Installation:
- Move the attached file titled wcp_cert_manager.zip to the jumpbox where vSphere plugin for kubectl is installed or Supervisor CP node. (Use WinSCP from Windows OS's if required):
Example Output:
The authenticity of host '192.168.0.2 (192.168.0.2)' can't be established.
ECDSA key fingerprint is SHA256:RkfHc8xvRJ8ihqMD1CTQeMXEPrYJ6yaNEOhwKpCbt3w.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.2' (ECDSA) to the list of known hosts.
VMware vCenter Server 7.0.3.01000
Type: vCenter Server with an embedded Platform Services Controller
Password:
wcp_cert_manager.zip 100% 8473KB 8.3MB/s 00:00
- Unzip the file and move it to the executable path:
# unzip wcp_cert_manager.zip
Archive: wcp_cert_manager.zip
inflating: certmgr
# ls -l
total 30956
-rwxr-xr-x 1 root root 23019418 Nov 28 01:24 certmgr
-rw-r--r-- 1 root root 8675846 Jan 17 16:09 wcp_cert_manager.zip
# cp certmgr /usr/bin/
Execution:
- List Guest Cluster certificates:
# certmgr tkc certificates list -n <NAMESPACE_NAME> <CLUSTER_NAME>
Example Output:
# certmgr tkc certificates list -n certs cluster1
20:53:04 proc.go:267: [/root/certmgr tkc certificates list -n certs cluster1]
20:53:04 list.go:20: checking certs on machine, kind: Machine, namespace: certs, name: cluster1-control-plane-v5n8k, ip: 10.244.0.10
20:53:04 client.go:196: copying certmgr to remote, kind: Machine, namespace: certs, name: cluster1-control-plane-v5n8k
20:53:05 scp.go:86: copying file certmgr to /home/vmware-system-user/certmgr with size 64 MiB, mode 750
Uploading 64 MiB/64 MiB
20:53:06 client.go:196: finished copying
/etc/bash.bashrc: line 43: TMOUT: readonly variable
/etc/bash.bashrc: line 43: TMOUT: readonly variable
20:53:06 proc.go:267: [/root/certmgr certificates list]
20:53:06 proc.go:267: program exited
+--------------+-------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
| SCOPE | IP | HOSTNAME | NAME | NOTAFTER | ISEXPIRED |
+--------------+-------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
| controlplane | 10.244.0.10 | cluster1-control-plane-v5n8k | /etc/kubernetes/pki/front-proxy-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/apiserver.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/apiserver-etcd-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/apiserver-kubelet-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet.crt | 2024-10-26 15:11:59 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet-client-current.pem | 2024-10-26 15:22:47 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/server.crt | 2024-09-17 15:02:58 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/peer.crt | 2024-10-26 15:32:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/healthcheck-client.crt | 2024-10-26 15:22:40 +0000 UTC | false |
| | | | /etc/kubernetes/pki/front-proxy-ca.crt | 2033-09-15 14:57:11 +0000 UTC | false |
| | | | /etc/kubernetes/pki/ca.crt | 2033-09-15 14:57:11 +0000 UTC | false |
| | | | /etc/kubernetes/pki/etcd/ca.crt | 2033-09-15 14:57:12 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet.crt | 2024-10-26 15:11:59 +0000 UTC | false |
| | | | /var/lib/kubelet/pki/kubelet-client-current.pem | 2024-10-26 15:22:47 +0000 UTC | false |
+--------------+-------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
20:53:06 list.go:54: command execution completed successfully.
20:53:06 proc.go:267: program exited
- Rotate Guest Cluster certificates:
# certmgr tkc certificates rotate -n <NAMESPACE_NAME> <CLUSTER_NAME>
- Restart kube-controller-manager pods
- On each control plane node, run the following:
- crictl rm -f $(crictl ps --label io.kubernetes.container.name=kube-controller-manager -q)
Example Output:
# certmgr tkc certificates rotate -n certs cluster1
Uploading 64 MiB/64 MiB
/etc/bash.bashrc: line 43: TMOUT: readonly variable
/etc/bash.bashrc: line 43: TMOUT: readonly variable
20:58:17 proc.go:267: [/root/certmgr certificates rotate]
20:58:22 etcd_actions.go:66: etcd healthy after 0.83 seconds
20:58:22 root.go:265: result {[{backup certificates /root } {rotate etcd server certificate true } {rotate api server etcd client certificate true } {rotate etcd peer certificate true } {rotate etcd health check certificate true } {rotate api server certificate true } {rotate kubelet client api server certificate true } {rotate front proxy certificate true } {rotate controller-manager certificate true } {rotate scheduler certificate true } {rotate kubelet certificate <nil> } {rotate kubeadm admin certificate true } {verify etcd health true }] ok <nil>}
20:58:22 proc.go:267: program exited
+-----------------------------------------------------+----------------+
| TASKS | OVERALL STATUS |
+-----------------------------------------------------+----------------+
| +--------------------------------+--------+-------+ | ok |
| | TASK | RESULT | ERROR | | |
| +--------------------------------+--------+-------+ | |
| | backup certificates | /root | | | |
| | rotate etcd server certificate | true | | | |
| | rotate api server etcd client | true | | | |
| | certificate | | | | |
| | rotate etcd peer certificate | true | | | |
| | rotate etcd health check | true | | | |
| | certificate | | | | |
| | rotate api server certificate | true | | | |
| | rotate kubelet client api | true | | | |
| | server certificate | | | | |
| | rotate front proxy certificate | true | | | |
| | rotate controller-manager | true | | | |
| | certificate | | | | |
| | rotate scheduler certificate | true | | | |
| | rotate kubelet certificate | | | | |
| | rotate kubeadm admin | true | | | |
| | certificate | | | | |
| | verify etcd health | true | | | |
| +--------------------------------+--------+-------+ | |
| | |
+-----------------------------------------------------+----------------+
NOTE:
- This tool replaces ONLY the ControlPlane certificates in Guest Clusters currently. Worker node cert replacement must be applied with a worker node rollout.
- All logs for this tool are logged under /var/log/vmware/certmgr.log