Replace vSphere with Tanzu Supervisor Certificates

Products

VMware vCenter Server

Issue/Introduction

vSphere with Tanzu supervisor certificates or spherelet certificates have expired or are about to expire.

If you need to rotate guest cluster cert's please use the following kb: https://knowledge.broadcom.com/external/article?articleId=323453

You can use the following command while connected via SSH into either of the Supervisor Control Plane VMs.

# find / -type f $ -name "*.cert" -o -name "*.crt" $ -print 2>/dev/null | egrep -iv 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd|backup' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'

SSH into Supervisor Control Plane VM Instructions: Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's (90194)

Environment

VMware vCenter Server 7.0.x

Resolution

Install the wcp_cert_manager tool to vCenter

Move the attached file titled wcp_cert_manager.zip to the vCenter Server (under /root/ ) where vSphere with Tanzu is deployed. (Use WinSCP from Windows OS's if required):

$ scp ./wcp_cert_manager.zip root@##.##.##.##:/root

Example Output:

The authenticity of host '##.##.##.## (##.##.##.##)' can't be established.
ECDSA key fingerprint is SHA256:<fingerprint>.

Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '##.##.##.##' (ECDSA) to the list of known hosts.

VMware vCenter Server 7.0.#.#####

Type: vCenter Server with an embedded Platform Services Controller

Password:
wcp_cert_manager.zip 100% 8473KB 8.3MB/s 00:00

Make sure you are under /root and unzip the file:

# pwd
/root

# unzip wcp_cert_manager.zip

Archive: wcp_cert_manager.zip
inflating: certmgr

# ls -l

total 30956
-rwxr-xr-x 1 root root 23019418 MM DD HH:MM certmgr
-rw-r--r-- 1 root root 8675846 MM DD HH:MM wcp_cert_manager.zip

Using the wcp_cert_manager tool

From /root run './certmgr certificates rotate command' to rotate all supervisor control plane certificates and spherelet certificates. Below is an example of a successful cert rotation.

If you have multiple vSphere with Tanzu deployments on your vCenter, then you need to use the -c command to specify the cluster you want to replace certificates on.
In order to gather the cluster id you can run
./certmgr supervisors
ie
root@###### [ ~ ]# ./certmgr supervisors
YYYY/MM/DD HH:MM:DD Cluster: domain-c#:########-####-####-####-#############

IP: ##.##.##.##
Password: ***********************************

In the above example the cluster id would be "domain-c#:########-####-####-####-#############"

An example of running the tool on a specific cluster would be.
# ./certmgr certificates rotate -c domain-c#:########-####-####-####-#############

All logs for this tool are logged under /var/log/vmware/certmgr.log

Additional Information

Main vSphere with Tanzu Cert Page: vSphere with Tanzu Certificate Guide

In case the 'certmgr' tool returns no output or fails with below error, please make sure you are running the command from /root/ of the vCenter Server.

"Error running supervisor cert manager: error while running CPVM cmd: Error running cmd on (##.##.##.##): /tmp/certmgr certificates list --json, error: Process exited with status 127"

Attachments

wcp_cert_manager.zip get_app