Replace vSphere with Tanzu Guest Cluster Certificates
search cancel

Replace vSphere with Tanzu Guest Cluster Certificates

book

Article ID: 323453

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere with Tanzu

Issue/Introduction

Symptoms:
  • vSphere with Tanzu Guest Cluster certificates have expired or are about to expire. 
  • Use the following command while connected via SSH into either of the Guest Cluster Control Plane VMs. 
 
# find / -type f \( -name "*.cert" -o -name "*.crt" \) -print 2>/dev/null | egrep -iv 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd|backup' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'
 

Environment

VMware vSphere 8.0 with Tanzu
VMware vSphere 7.0 with Tanzu

Resolution

Prerequisites:


The attached wcp_cert_manager tool can be run from either of two locations to replace Guest Cluster certificates:
 

1. From a jumpbox that has the kubectl and vSphere Plugin for kubectl installed that also has network connectivity to the Workload Network. Instructions for download and install of these utilities can be found here
2. From an SSH session to one of the Supervisor ControlPlane nodes. SSH into Supervisor Control Plane VM Instructions: Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's (90194)
 


Installation:
 
  • Move the attached file titled wcp_cert_manager.zip to the jumpbox where vSphere plugin for kubectl is installed or Supervisor CP node. (Use WinSCP from Windows OS's if required): 

# scp ./wcp_cert_manager.zip [email protected]:/root
 
 
Example Output:
 
The authenticity of host '192.168.0.2 (192.168.0.2)' can't be established.
ECDSA key fingerprint is SHA256:RkfHc8xvRJ8ihqMD1CTQeMXEPrYJ6yaNEOhwKpCbt3w.
 
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.2' (ECDSA) to the list of known hosts.
 
VMware vCenter Server 7.0.3.01000
 
Type: vCenter Server with an embedded Platform Services Controller
 
Password:
wcp_cert_manager.zip                                                                                                                                    100% 8473KB   8.3MB/s   00:00
 
 
  • Unzip the file and move it to the executable path:
 
# unzip wcp_cert_manager.zip
Archive:  wcp_cert_manager.zip
  inflating: certmgr
 
# ls -l
total 30956
-rwxr-xr-x 1 root root 23019418 Nov 28 01:24 certmgr
-rw-r--r-- 1 root root  8675846 Jan 17 16:09 wcp_cert_manager.zip

# cp certmgr /usr/bin/
 


Execution:
 
  • List Guest Cluster certificates:
certmgr tkc certificates list -n <NAMESPACE_NAME> <CLUSTER_NAME>
 
Example Output:
 
certmgr tkc certificates list -n certs cluster1
20:53:04 proc.go:267: [/root/certmgr tkc certificates list -n certs cluster1]
20:53:04 list.go:20: checking certs on machine, kind: Machine, namespace: certs, name: cluster1-control-plane-v5n8k, ip: 10.244.0.10
20:53:04 client.go:196: copying certmgr to remote, kind: Machine, namespace: certs, name: cluster1-control-plane-v5n8k
20:53:05 scp.go:86: copying file certmgr to /home/vmware-system-user/certmgr with size 64 MiB, mode 750
Uploading 64 MiB/64 MiB
20:53:06 client.go:196: finished copying
/etc/bash.bashrc: line 43: TMOUT: readonly variable
/etc/bash.bashrc: line 43: TMOUT: readonly variable
20:53:06 proc.go:267: [/root/certmgr certificates list]
20:53:06 proc.go:267: program exited
+--------------+-------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
|    SCOPE     |     IP      |         HOSTNAME          |                       NAME                       |           NOTAFTER            | ISEXPIRED |
+--------------+-------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
| controlplane | 10.244.0.10 | cluster1-control-plane-v5n8k | /etc/kubernetes/pki/front-proxy-client.crt       | 2024-10-26 15:22:40 +0000 UTC | false     |
|              |             |                           | /etc/kubernetes/pki/apiserver.crt                | 2024-10-26 15:22:40 +0000 UTC | false     |
|              |             |                           | /etc/kubernetes/pki/apiserver-etcd-client.crt    | 2024-10-26 15:22:40 +0000 UTC | false     |
|              |             |                           | /etc/kubernetes/pki/apiserver-kubelet-client.crt | 2024-10-26 15:22:40 +0000 UTC | false     |
|              |             |                           | /var/lib/kubelet/pki/kubelet.crt                 | 2024-10-26 15:11:59 +0000 UTC | false     |
|              |             |                           | /var/lib/kubelet/pki/kubelet-client-current.pem  | 2024-10-26 15:22:47 +0000 UTC | false     |
|              |             |                           | /etc/kubernetes/pki/etcd/server.crt              | 2024-09-17 15:02:58 +0000 UTC | false     |
|              |             |                           | /etc/kubernetes/pki/etcd/peer.crt                | 2024-10-26 15:32:40 +0000 UTC | false     |
|              |             |                           | /etc/kubernetes/pki/etcd/healthcheck-client.crt  | 2024-10-26 15:22:40 +0000 UTC | false     |
|              |             |                           | /etc/kubernetes/pki/front-proxy-ca.crt           | 2033-09-15 14:57:11 +0000 UTC | false     |
|              |             |                           | /etc/kubernetes/pki/ca.crt                       | 2033-09-15 14:57:11 +0000 UTC | false     |
|              |             |                           | /etc/kubernetes/pki/etcd/ca.crt                  | 2033-09-15 14:57:12 +0000 UTC | false     |
|              |             |                           | /var/lib/kubelet/pki/kubelet.crt                 | 2024-10-26 15:11:59 +0000 UTC | false     |
|              |             |                           | /var/lib/kubelet/pki/kubelet-client-current.pem  | 2024-10-26 15:22:47 +0000 UTC | false     |
+--------------+-------------+---------------------------+--------------------------------------------------+-------------------------------+-----------+
20:53:06 list.go:54: command execution completed successfully.
20:53:06 proc.go:267: program exited
 
 
  • Rotate Guest Cluster certificates:
certmgr tkc certificates rotate -n <NAMESPACE_NAME> <CLUSTER_NAME>
  • Restart kube-controller-manager pods
  • On each control plane node, run the following:
    • crictl rm -f $(crictl ps --label io.kubernetes.container.name=kube-controller-manager -q)
 

Example Output:
 
certmgr tkc certificates rotate -n certs cluster1
Uploading 64 MiB/64 MiB
/etc/bash.bashrc: line 43: TMOUT: readonly variable
/etc/bash.bashrc: line 43: TMOUT: readonly variable
20:58:17 proc.go:267: [/root/certmgr certificates rotate]
20:58:22 etcd_actions.go:66: etcd healthy after 0.83 seconds
20:58:22 root.go:265: result {[{backup certificates /root } {rotate etcd server certificate true } {rotate api server etcd client certificate true } {rotate etcd peer certificate true } {rotate etcd health check certificate true } {rotate api server certificate true } {rotate kubelet client api server certificate true } {rotate front proxy certificate true } {rotate controller-manager certificate true } {rotate scheduler certificate true } {rotate kubelet certificate <nil> } {rotate kubeadm admin certificate true } {verify etcd health true }] ok <nil>}
20:58:22 proc.go:267: program exited
+-----------------------------------------------------+----------------+
|                        TASKS                        | OVERALL STATUS |
+-----------------------------------------------------+----------------+
| +--------------------------------+--------+-------+ | ok             |
| |              TASK              | RESULT | ERROR | |                |
| +--------------------------------+--------+-------+ |                |
| | backup certificates            | /root  |       | |                |
| | rotate etcd server certificate | true   |       | |                |
| | rotate api server etcd client  | true   |       | |                |
| | certificate                    |        |       | |                |
| | rotate etcd peer certificate   | true   |       | |                |
| | rotate etcd health check       | true   |       | |                |
| | certificate                    |        |       | |                |
| | rotate api server certificate  | true   |       | |                |
| | rotate kubelet client api      | true   |       | |                |
| | server certificate             |        |       | |                |
| | rotate front proxy certificate | true   |       | |                |
| | rotate controller-manager      | true   |       | |                |
| | certificate                    |        |       | |                |
| | rotate scheduler certificate   | true   |       | |                |
| | rotate kubelet certificate     |        |       | |                |
| | rotate kubeadm admin           | true   |       | |                |
| | certificate                    |        |       | |                |
| | verify etcd health             | true   |       | |                |
| +--------------------------------+--------+-------+ |                |
|                                                     |                |
+-----------------------------------------------------+----------------+

 
 
NOTE:
  • This tool replaces ONLY the ControlPlane certificates in Guest Clusters currently. Worker node cert replacement must be applied with a worker node rollout. 
  • All logs for this tool are logged under /var/log/vmware/certmgr.log 


Additional Information

Main vSphere with Tanzu Cert Page: https://knowledge.broadcom.com/external/article?legacyId=89324

Attachments

wcp_cert_manager get_app