Replace vSphere with Tanzu Supervisor Certificates
Article ID: 322994
Updated On:
Products
VMware vCenter Server
VMware vSphere Kubernetes Service
Tanzu Kubernetes Runtime
Issue/Introduction
vSphere with Tanzu Supervisor Cluster certificates or ESXi spherelet certificates have expired or are about to expire.
The following commands can be used to check the expiration of Kubernetes certificates on the Supervisor cluster:
While SSH to each Supervisor control plane VM:
#Checks for wcp/tls and kubernetes/pki certificates
find /etc -type f \( -name "*.cert" -o -name "*.crt" \) | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'
#Check for the kubelet certificate's expiration
openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet.crt | grep After
#Checks that the admin.conf file's certificates are not expired
cat /etc/kubernetes/admin.conf | grep certificate-authority-data | awk '{print $2}' | base64 -d | openssl x509 -noout -text | grep After
cat /etc/kubernetes/admin.conf | grep client-certificate-data | awk '{print $2}' | base64 -d | openssl x509 -noout -text | grep After
The attached certmgr script can be used to check for certificate expiration dates with the following commands when run from the VCSA:
#Retrieve the ID of the Supervisor Cluster to check its certificates
./certmgr supervisors
#Use certmgr to list the certificates for the Supervisor Cluster
./certmgr certificates list -c <domain id for the supervisor cluster>
Workload/Guest cluster certificates use the same certmgr script when run from a jumpbox with kubectl CLI or from the Supervisor cluster.
The instructions can be found in the following KB:
Environment
vSphere Supervisor 7
vSphere Supervisor 8
vSphere Supervsior 9
Cause
Kubernetes certificates are set to expire after a year by default. VMware by Broadcom Kubernetes products adhere to this certificate expiry timeline.
Resolution
NOTE: Before running the attached certmgr tool, check the below items to make sure of the following:
- Time synchronized across the environment, such as vCenter Server, ESXi, Supervisor Control Plane VMs.
- Supervisor '
Config Status' isRunningand 'Host Config Status' isRunning. See Workload Management configuration monitoring - All nodes in Supervisor cluster are
Ready. See Connect to the Supervisor Cluster as a vCenter Single Sign-On User
Install the wcp_cert_manager tool to vCenter
- Move the attached file titled wcp_cert_manager.zip to the vCenter Server (under /root/ ) where vSphere with Tanzu is deployed. (Use WinSCP from Windows OS if required):
$ scp ./wcp_cert_manager.zip root@##.##.##.##:/root The authenticity of host '##.##.##.## (##.##.##.##)' can't be established. ECDSA key fingerprint is SHA256:<fingerprint>. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '##.##.##.##' (ECDSA) to the list of known hosts. VMware vCenter Server #.0.#.##### Type: vCenter Server with an embedded Platform Services Controller Password: wcp_cert_manager.zip 100% 8473KB 8.3MB/s 00:00If you encounter an issue using WinSCP, please see the following KB article:
Connecting to vCenter Server Virtual Appliance using WinSCP fails - Make sure you are under /root and unzip the file:
pwd /root unzip wcp_cert_manager.zip Archive: wcp_cert_manager.zip inflating: certmgr ls -l total 30956 -rwxr-xr-x 1 root root 23019418 MM DD HH:MM certmgr -rw-r--r-- 1 root root 8675846 MM DD HH:MM wcp_cert_manager.zip
Using the wcp_cert_manager tool
- From /root run './certmgr certificates rotate' command to rotate all supervisor control plane certificates and spherelet certificates.
Below is an example of a successful cert rotation:./certmgr certificates rotate +------------------+------------------------------------------------------------------------------------------------------+-------+ | CONTROL PLANE IP | RESULT | ERROR | +------------------+------------------------------------------------------------------------------------------------------+-------+ | ##.##.##.## | +---------------------------------------------------------------------------------+----------------+ | | | | | TASKS | OVERALL STATUS | | | | | +---------------------------------------------------------------------------------+----------------+ | | | | | +--------------------------------+------------------------------------+-------+ | ok | | | | | | | TASK | RESULT | ERROR | | | | | | | | +--------------------------------+------------------------------------+-------+ | | | | | | | | backup certificates | /root/backups-16739895901776834456 | | | | | | | | | | rotate etcd server certificate | | | | | | | | | | | rotate api server etcd client | | | | | | | | | | | certificate | | | | | | | | | | | rotate etcd peer certificate | | | | | | | | | | | rotate etcd health check | | | | | | | | | | | certificate | | | | | | | | | | | rotate api server certificate | | | | | | | | | | | rotate kubelet client api | | | | | | | | | | | server certificate | | | | | | | | | | | rotate front proxy certificate | | | | | | | | | | | rotate controller-manager | | | | | | | | | | | certificate | | | | | | | | | | | rotate scheduler certificate | | | | | | | | | | | rotate scheduler extension | | | | | | | | | | | certificate | | | | | | | | | | | rotate kubelet certificate | | | | | | | | | | | restart ncp | NCP restart skipped: VDS setup | | | | | | | | | | | detected. | | | | | | | | | | rotate auth proxy certificate | | | | | | | | | | | rotate management certificate | | | | | | | | | | | rotate registry certificate | | | | | | | | | | | rotate kubeadm admin | | | | | | | | | | | certificate | | | | | | | | | | | verify etcd health | | | | | | | | | | +--------------------------------+------------------------------------+-------+ | | | | | | | | | | | | | +---------------------------------------------------------------------------------+----------------+ | | | | | | | ##.##.##.## | +---------------------------------------------------------------------------------+----------------+ | | | | | TASKS | OVERALL STATUS | | | | | +---------------------------------------------------------------------------------+----------------+ | | | | | +--------------------------------+------------------------------------+-------+ | ok | | | | | | | TASK | RESULT | ERROR | | | | | | | | +--------------------------------+------------------------------------+-------+ | | | | | | | | backup certificates | /root/backups-16739895893751688144 | | | | | | | | | | rotate etcd server certificate | | | | | | | | | | | rotate api server etcd client | | | | | | | | | | | certificate | | | | | | | | | | | rotate etcd peer certificate | | | | | | | | | | | rotate etcd health check | | | | | | | | | | | certificate | | | | | | | | | | | rotate api server certificate | | | | | | | | | | | rotate kubelet client api | | | | | | | | | | | server certificate | | | | | | | | | | | rotate front proxy certificate | | | | | | | | | | | rotate controller-manager | | | | | | | | | | | certificate | | | | | | | | | | | rotate scheduler certificate | | | | | | | | | | | rotate scheduler extension | | | | | | | | | | | certificate | | | | | | | | | | | rotate kubelet certificate | | | | | | | | | | | restart ncp | NCP restart skipped: VDS setup | | | | | | | | | | | detected. | | | | | | | | | | rotate auth proxy certificate | | | | | | | | | | | rotate management certificate | | | | | | | | | | | rotate registry certificate | | | | | | | | | | | rotate kubeadm admin | | | | | | | | | | | certificate | | | | | | | | | | | verify etcd health | | | | | | | | | | +--------------------------------+------------------------------------+-------+ | | | | | | | | | | | | | +---------------------------------------------------------------------------------+----------------+ | | | | | | | ##.##.##.## | +--------------------------------------------------------------------------------+----------------+ | | | | | TASKS | OVERALL STATUS | | | | | +--------------------------------------------------------------------------------+----------------+ | | | | | +--------------------------------+-----------------------------------+-------+ | ok | | | | | | | TASK | RESULT | ERROR | | | | | | | | +--------------------------------+-----------------------------------+-------+ | | | | | | | | backup certificates | /root/backups-1673989589793637456 | | | | | | | | | | rotate etcd server certificate | | | | | | | | | | | rotate api server etcd client | | | | | | | | | | | certificate | | | | | | | | | | | rotate etcd peer certificate | | | | | | | | | | | rotate etcd health check | | | | | | | | | | | certificate | | | | | | | | | | | rotate api server certificate | | | | | | | | | | | rotate kubelet client api | | | | | | | | | | | server certificate | | | | | | | | | | | rotate front proxy certificate | | | | | | | | | | | rotate controller-manager | | | | | | | | | | | certificate | | | | | | | | | | | rotate scheduler certificate | | | | | | | | | | | rotate scheduler extension | | | | | | | | | | | certificate | | | | | | | | | | | rotate kubelet certificate | | | | | | | | | | | restart ncp | NCP restart skipped: NCP | | | | | | | | | | | restart only occurs on the | | | | | | | | | | | leader. | | | | | | | | | | rotate auth proxy certificate | | | | | | | | | | | rotate management certificate | | | | | | | | | | | rotate registry certificate | | | | | | | | | | | rotate kubeadm admin | | | | | | | | | | | certificate | | | | | | | | | | | verify etcd health | | | | | | | | | | +--------------------------------+-----------------------------------+-------+ | | | | | | | | | | | | | +--------------------------------------------------------------------------------+----------------+ | | | | | | +------------------+------------------------------------------------------------------------------------------------------+-------+ +-----------------------------------------------------+----------------+ | TASKS | OVERALL STATUS | +-----------------------------------------------------+----------------+ | +--------------------------------+--------+-------+ | | | | TASK | RESULT | ERROR | | | | +--------------------------------+--------+-------+ | | | | rotate spherelet certificates | | | | | | | on ##.##.##.## (host-##) | | | | | | | rotate spherelet certificates | | | | | | | on ##.##.##.## (host-##) | | | | | | | rotate spherelet certificates | | | | | | | on ##.##.##.## (host-##) | | | | | | +--------------------------------+--------+-------+ | | | | | +-----------------------------------------------------+----------------+
If you have multiple vSphere with Tanzu deployments on your vCenter, then you need to use the -c argument to specify the cluster you want to replace certificates on.
In order to gather the supervisor cluster id you can run:
./certmgr supervisors
./certmgr supervisors
YYYY/MM/DD HH:MM:DD Cluster: domain-c#:########-####-####-####-#############
IP: ##.##.##.##
Password: ***********************************
In the above example the cluster id would be "domain-c#:########-####-####-####-#############"
An example of running the tool on a specific cluster would be:
./certmgr certificates list -c domain-c#:########-####-####-####-#############
./certmgr certificates rotate -c domain-c#:########-####-####-####-#############
To ensure the rotated certificate keys are persisted upon reboot:
- SSH into each Supervisor control plane VM:
See SSH into Supervisor Control Plane VM Instructions: Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's (323407) - Perform the following script command on each Supervisor control plane VM:
/usr/lib/vmware-wcp/hypercrypt.py --reencrypt
Note: In a vSphere 7.0 environment, sincehypercrypt.pylacks thereencryptoption, createreencrypt.pyas a substitute and run that Python script:
- Create reencrypt.py as follows.
# cat <<EOF>> reencrypt.py import os from subprocess import run import yaml fh=open("/etc/vmware/wcp/secrets-config.yaml") config=yaml.load(fh, Loader=yaml.FullLoader) for file_id, file_info in config["files"].items(): if isinstance(file_info, dict): path = file_info["path"] else: path = file_info if os.path.exists(path): run(["/usr/lib/vmware-wcp/hypercrypt.py", "--file-name", file_id, "--read", path]) EOF # Press Ctrl+D to finish editing reencrypt.py. - Run reencrypt.py
python ./reencrypt.py
- Create reencrypt.py as follows.
Additional Information
Troubleshooting Certmgr
All logs for this tool are logged in the VCSA under /var/log/vmware/certmgr.log
In case the 'certmgr' tool returns no output or fails with below error, ensure that you are running the command from /root/ of the vCenter Server.
"Error running supervisor cert manager: error while running CPVM cmd: Error running cmd on (##.##.##.##): /tmp/certmgr certificates list --json, error: Process exited with status 127"
Admin.Conf Certificates
Note - Nov 14, 2025: The certmgr tool had an issue where it does not always successfully rotate the admin.conf 's certificates. This should be resolved in v0.0.19, but if the issue is observed, please open a ticket with Broadcom Support to troubleshoot and fix the script.
When these admin.conf certificates are expired, kubectl commands will fail.
See the following KB article to manually rotate the admin.conf certificates:
Unable to run kubectl command on a Supervisor cluster CPVM with "error: You must be logged in to the server (Unauthorized)"
ESXi Host Not Ready - Spherelet Certificates
If the ESXi hosts show Not Ready state after running the 'certmgr' tool, see ESXi nodes become NotReady after rotating Supervisor Certificates using certmgr
Kubelet.Conf Certificates
Note - Nov 14,2025: The certmgr tool had an issue where it did not always successfully rotate the kubelet.conf 's embedded certificate. This should be resolved in v0.0.19, but if the issue is observed, please open a ticket with Broadcom Support to troubleshoot and fix the script.
See the following KB article to manually rotate the embedded certificate in kubelet.conf:
Supervisor controlPlane Node NotReady Error "unable to load bootstrap kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory"
Notes on certificates and keys that are not renewed by certmgr:
- Main vSphere with Tanzu Certificate KB: vSphere with Tanzu Certificate Guide
- /etc/vmware/wcp/tls/authproxy-client.crt and /etc/vmware/wcp/tls/pinniped.crt certificates on the Supervisor control plane VMs are not renewed by the certmgr script.
- These certs are auto-regenerated when their expiration hits 2/3 of their overall lifetime.
- If these certificates have expired, please open a ticket with VMware by Broadcom support for assistance in regenerating them.
- These two certificates cannot be replaced with custom certificates.
- Supervisor cluster system pod certificates are not managed by the certmgr script. These certificates are managed instead by the cert-manager system pod in the Supervisor cluster and are expected to automatically renew before expiry. If these certificates are found to be expired, see the following KB:
vSphere Supervisor System Pod Certificate Expiry due to Cert-Manager Issues - Older versions of certmgr tools do not rotate spherelet certificates, which will leave Supervisor Clusters worker nodes (ESXi hosts) in 'Not Ready' state and Host Config in Configuring state. It is important to use the latest certmgr tool from this KB.
Updates
v0.0.19
Nov 12,2025
- Enforce TLS to use FIPS approved settings
- Added subtask to restart wcp-fip pod after the cert-rotation
- Handle apiserver-proxy server/client certs rotation and removes supervisor-nginx cert
- Added retry while failed to stop the container
- Improve list command to print client cert expiry from config files
- Add logic in spherelet cert generation code to take timezone difference in account
- Add steps to restart static pods after hypercrypt runs.
- Add appplatform authz cert rotation
- Added kubeconfig in kubectl commands
- Container restart for appplatform authorization webhook after cert rotation
- Add cert rotation for apiserver webhook cert
- Support cert rotation for mgmt-image-proxy.crt and supervisor-nginx.crt
- Added step to reencrypt keys using hypercrypt after rotation
Attachments
Feedback
Yes
No
Powered by
