Replace vSphere with Tanzu Supervisor Certificates

Products

VMware vCenter Server

Issue/Introduction

vSphere with Tanzu supervisor certificates or spherelet certificates have expired or are about to expire.

You can use the following command while connected via SSH into either of the Supervisor Control Plane VMs.

# find / -type f $ -name "*.cert" -o -name "*.crt" $ -print 2>/dev/null | egrep -iv 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd|backup' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'

SSH into Supervisor Control Plane VM Instructions: Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's (90194)

Environment

VMware vCenter Server 7.0.x

Resolution

Install the wcp_cert_manager tool to vCenter

Move the attached file titled wcp_cert_manager.zip to the vCenter Server (under /root/ ) where vSphere with Tanzu is deployed. (Use WinSCP from Windows OS's if required):

$ scp ./wcp_cert_manager.zip root@##.##.##.##:/root

Example Output:

The authenticity of host '##.##.##.## (##.##.##.##)' can't be established.
ECDSA key fingerprint is SHA256:<fingerprint>.

Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '##.##.##.##' (ECDSA) to the list of known hosts.

VMware vCenter Server 7.0.#.#####

Type: vCenter Server with an embedded Platform Services Controller

Password:
wcp_cert_manager.zip 100% 8473KB 8.3MB/s 00:00

Make sure you are under /root and unzip the file:

# pwd
/root

# unzip wcp_cert_manager.zip

Archive: wcp_cert_manager.zip
inflating: certmgr

# ls -l

total 30956
-rwxr-xr-x 1 root root 23019418 MM DD HH:MM certmgr
-rw-r--r-- 1 root root 8675846 MM DD HH:MM wcp_cert_manager.zip

Using the wcp_cert_manager tool

From /root run './certmgr certificates rotate command' to rotate all supervisor control plane certificates and spherelet certificates. Below is an example of a successful cert rotation.

If you have multiple vSphere with Tanzu deployments on your vCenter, then you need to use the -c command to specify the cluster you want to replace certificates on.
In order to gather the cluster id you can run
./certmgr supervisors
ie
root@###### [ ~ ]# ./certmgr supervisors
YYYY/MM/DD HH:MM:DD Cluster: domain-c#:########-####-####-####-#############

IP: ##.##.##.##
Password: ***********************************

In the above example the cluster id would be "domain-c#:########-####-####-####-#############"

An example of running the tool on a specific cluster would be.
# ./certmgr certificates rotate -c domain-c#:########-####-####-####-#############

All logs for this tool are logged under /var/log/vmware/certmgr.log

Additional Information

Main vSphere with Tanzu Cert Page: vSphere with Tanzu Certificate Guide

In case the 'certmgr' tool returns no output or fails with below error, please make sure you are running the command from /root/ of the vCenter Server.

"Error running supervisor cert manager: error while running CPVM cmd: Error running cmd on (##.##.##.##): /tmp/certmgr certificates list --json, error: Process exited with status 127"

Attachments

wcp_cert_manager get_app