Replace vSphere with Tanzu / vSphere Kubernetes Service Supervisor Certificates
Article ID: 322994
Updated On:
Products
VMware vCenter Server
VMware vSphere Kubernetes Service
Tanzu Kubernetes Runtime
Issue/Introduction
- vSphere with Tanzu / vSphere Kubernetes Service Supervisor Cluster certificates or ESXi spherelet certificates have expired or are about to expire.
- The following commands can be used to check the expiration of Kubernetes certificates on the Supervisor cluster while using SSH to access each Supervisor control plane VM:
#Checks for wcp/tls and kubernetes/pki certificates find /etc -type f \( -name "*.cert" -o -name "*.crt" \) | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After' #Check for the kubelet certificate's expiration openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet.crt | grep After #Checks that the admin.conf file's certificates are not expired cat /etc/kubernetes/admin.conf | grep certificate-authority-data | awk '{print $2}' | base64 -d | openssl x509 -noout -text | grep After cat /etc/kubernetes/admin.conf | grep client-certificate-data | awk '{print $2}' | base64 -d | openssl x509 -noout -text | grep After
- The attached certmgr script can be used to check for certificate expiration dates with the following commands when run from the VCSA:
#Retrieve the ID of the Supervisor Cluster to check its certificates ./certmgr supervisors #Use certmgr to list the certificates for the Supervisor Cluster ./certmgr certificates list -c <domain id for the supervisor cluster> Example output of the above command: /root/certmgr certificates list time="2025-12-03T09:27:59Z" level=info msg="[/root/certmgr certificates list]" +--------------+--------------+----------------------------------+------------------------------------------------------------------+-------------------------------+-----------+ | SCOPE | IP | HOSTNAME | NAME | NOTAFTER | ISEXPIRED | +--------------+--------------+----------------------------------+------------------------------------------------------------------+-------------------------------+-----------+ | controlplane | ##.###.##.## | 420fb95c6d8b291ef418ad7d4d3f07c2 | /etc/kubernetes/admin.conf(client-certificate-data) | 2026-12-03 07:08:42 +0000 UTC | false | | | | | /etc/kubernetes/controller-manager.conf(client-certificate-data) | 2026-12-03 07:08:42 +0000 UTC | false | | | | | /etc/kubernetes/pki/front-proxy-client.crt | 2026-12-03 07:08:42 +0000 UTC | false | | | | | /etc/kubernetes/pki/apiserver.crt | 2026-12-03 07:08:42 +0000 UTC | false | | | | | /etc/kubernetes/pki/apiserver-etcd-client.crt | 2026-12-03 07:08:42 +0000 UTC | false |
- Workload/Guest cluster certificates use the same certmgr script when run from a jumpbox with kubectl CLI or from the Supervisor cluster. The instructions can be found in the following KB:
Replace vSphere with Tanzu Guest Cluster Certificates
Cause
Kubernetes certificates are set to expire after a year by default. VMware by Broadcom Kubernetes products adhere to this certificate expiry timeline.
Resolution
NOTE: Before running the attached certmgr tool, check the below items to make sure of the following:
- Time is synchronized across the environment, such as vCenter Server, ESXi, Supervisor Control Plane VMs.
- Supervisor '
Config Status' isRunningand 'Host Config Status' isRunning. See Workload Management configuration monitoring - All nodes in Supervisor cluster are
Ready. See Connect to the Supervisor Cluster as a vCenter Single Sign-On User
Install the wcp_cert_manager tool to vCenter
- Move the attached file titled wcp_cert_manager.zip to the vCenter Server (under
/root/) where vSphere with Tanzu is deployed. (Use WinSCP from Windows OS if required):
$ scp ./wcp_cert_manager.zip root@##.##.##.##:/root The authenticity of host '##.##.##.## (##.##.##.##)' can't be established. ECDSA key fingerprint is SHA256:<fingerprint>. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '##.##.##.##' (ECDSA) to the list of known hosts. VMware vCenter Server #.0.#.##### Type: vCenter Server with an embedded Platform Services Controller Password: wcp_cert_manager.zip 100% 8473KB 8.3MB/s 00:00
If you encounter an issue using WinSCP, see the following KB article: Connecting to vCenter Server Virtual Appliance using WinSCP fails - Make sure you are under
/rootand unzip the file:
pwd
/root
unzip wcp_cert_manager.zip
Archive: wcp_cert_manager.zip
inflating: certmgr
ls -l
total 30956
-rwxr-xr-x 1 root root 23019418 MM DD HH:MM certmgr
-rw-r--r-- 1 root root 8675846 MM DD HH:MM wcp_cert_manager.zipUsing the wcp_cert_manager tool
- From /root run './certmgr certificates rotate' command to rotate all supervisor control plane certificates and spherelet certificates.
Below is an example of a successful cert rotation:
./certmgr certificates rotate +------------------+------------------------------------------------------------------------------------------------------+-------+ | CONTROL PLANE IP | RESULT | ERROR | +------------------+------------------------------------------------------------------------------------------------------+-------+ | ##.##.##.## | +---------------------------------------------------------------------------------+----------------+ | | | | | TASKS | OVERALL STATUS | | | | | +---------------------------------------------------------------------------------+----------------+ | | | | | +--------------------------------+------------------------------------+-------+ | ok | | | | | | | TASK | RESULT | ERROR | | | | | | | | +--------------------------------+------------------------------------+-------+ | | | | | | | | backup certificates | /root/backups-16739895901776834456 | | | | | | | | | | rotate etcd server certificate | | | | | | | | | | | rotate api server etcd client | | | | | | | | | | | certificate | | | | | | | | | | | rotate etcd peer certificate | | | | | | | | | | | rotate etcd health check | | | | | | | | | | | certificate | | | | | | | | | | | rotate api server certificate | | | | | | | | | | | rotate kubelet client api | | | | | | | | | | | server certificate | | | | | | | | | | | rotate front proxy certificate | | | | | | | | | | | rotate controller-manager | | | | | | | | | | | certificate | | | | | | | | | | | rotate scheduler certificate | | | | | | | | | | | rotate scheduler extension | | | | | | | | | | | certificate | | | | | | | | | | | rotate kubelet certificate | | | | | | | | | | | restart ncp | NCP restart skipped: VDS setup | | | | | | | | | | | detected. | | | | | | | | | | rotate auth proxy certificate | | | | | | | | | | | rotate management certificate | | | | | | | | | | | rotate registry certificate | | | | | | | | | | | rotate kubeadm admin | | | | | | | | | | | certificate | | | | | | | | | | | verify etcd health | | | | | | | | | | +--------------------------------+------------------------------------+-------+ | | | | | | | | | | | | | +---------------------------------------------------------------------------------+----------------+ | | | | | | | ##.##.##.## | +---------------------------------------------------------------------------------+----------------+ | | | | | TASKS | OVERALL STATUS | | | | | +---------------------------------------------------------------------------------+----------------+ | | | | | +--------------------------------+------------------------------------+-------+ | ok | | | | | | | TASK | RESULT | ERROR | | | | | | | | +--------------------------------+------------------------------------+-------+ | | | | | | | | backup certificates | /root/backups-16739895893751688144 | | | | | | | | | | rotate etcd server certificate | | | | | | | | | | | rotate api server etcd client | | | | | | | | | | | certificate | | | | | | | | | | | rotate etcd peer certificate | | | | | | | | | | | rotate etcd health check | | | | | | | | | | | certificate | | | | | | | | | | | rotate api server certificate | | | | | | | | | | | rotate kubelet client api | | | | | | | | | | | server certificate | | | | | | | | | | | rotate front proxy certificate | | | | | | | | | | | rotate controller-manager | | | | | | | | | | | certificate | | | | | | | | | | | rotate scheduler certificate | | | | | | | | | | | rotate scheduler extension | | | | | | | | | | | certificate | | | | | | | | | | | rotate kubelet certificate | | | | | | | | | | | restart ncp | NCP restart skipped: VDS setup | | | | | | | | | | | detected. | | | | | | | | | | rotate auth proxy certificate | | | | | | | | | | | rotate management certificate | | | | | | | | | | | rotate registry certificate | | | | | | | | | | | rotate kubeadm admin | | | | | | | | | | | certificate | | | | | | | | | | | verify etcd health | | | | | | | | | | +--------------------------------+------------------------------------+-------+ | | | | | | | | | | | | | +---------------------------------------------------------------------------------+----------------+ | | | | | | | ##.##.##.## | +--------------------------------------------------------------------------------+----------------+ | | | | | TASKS | OVERALL STATUS | | | | | +--------------------------------------------------------------------------------+----------------+ | | | | | +--------------------------------+-----------------------------------+-------+ | ok | | | | | | | TASK | RESULT | ERROR | | | | | | | | +--------------------------------+-----------------------------------+-------+ | | | | | | | | backup certificates | /root/backups-1673989589793637456 | | | | | | | | | | rotate etcd server certificate | | | | | | | | | | | rotate api server etcd client | | | | | | | | | | | certificate | | | | | | | | | | | rotate etcd peer certificate | | | | | | | | | | | rotate etcd health check | | | | | | | | | | | certificate | | | | | | | | | | | rotate api server certificate | | | | | | | | | | | rotate kubelet client api | | | | | | | | | | | server certificate | | | | | | | | | | | rotate front proxy certificate | | | | | | | | | | | rotate controller-manager | | | | | | | | | | | certificate | | | | | | | | | | | rotate scheduler certificate | | | | | | | | | | | rotate scheduler extension | | | | | | | | | | | certificate | | | | | | | | | | | rotate kubelet certificate | | | | | | | | | | | restart ncp | NCP restart skipped: NCP | | | | | | | | | | | restart only occurs on the | | | | | | | | | | | leader. | | | | | | | | | | rotate auth proxy certificate | | | | | | | | | | | rotate management certificate | | | | | | | | | | | rotate registry certificate | | | | | | | | | | | rotate kubeadm admin | | | | | | | | | | | certificate | | | | | | | | | | | verify etcd health | | | | | | | | | | +--------------------------------+-----------------------------------+-------+ | | | | | | | | | | | | | +--------------------------------------------------------------------------------+----------------+ | | | | | | +------------------+------------------------------------------------------------------------------------------------------+-------+ +-----------------------------------------------------+----------------+ | TASKS | OVERALL STATUS | +-----------------------------------------------------+----------------+ | +--------------------------------+--------+-------+ | | | | TASK | RESULT | ERROR | | | | +--------------------------------+--------+-------+ | | | | rotate spherelet certificates | | | | | | | on ##.##.##.## (host-##) | | | | | | | rotate spherelet certificates | | | | | | | on ##.##.##.## (host-##) | | | | | | | rotate spherelet certificates | | | | | | | on ##.##.##.## (host-##) | | | | | | +--------------------------------+--------+-------+ | | | | | +-----------------------------------------------------+----------------+ - If you have multiple vSphere with Tanzu deployments on your vCenter, then you need to use the -c argument to specify the cluster you want to replace certificates on.
In order to gather the supervisor cluster id you can run:
./certmgr supervisors ./certmgr supervisors YYYY/MM/DD HH:MM:DD Cluster: domain-c#:########-####-####-####-############# IP: ##.##.##.## Password: *********************************** In the above example the cluster id would be "domain-c#:########-####-####-####-#############" An example of running the tool on a specific cluster would be: ./certmgr certificates list -c domain-c#:########-####-####-####-############# ./certmgr certificates rotate -c domain-c#:########-####-####-####-############# - To ensure the rotated certificate keys are persisted upon reboot:
-
- SSH into each Supervisor control plane VM:
See SSH into Supervisor Control Plane VM Instructions: Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's (323407) - Perform the following script command on each Supervisor control plane VM:
/usr/lib/vmware-wcp/hypercrypt.py --reencrypt
- SSH into each Supervisor control plane VM:
NOTE: In a Legacy vSphere 7.0 environments, since hypercrypt.py lacks the reencrypt option, create reencrypt.py as a substitute and run that Python script:
-
-
Create reencrypt.py as follows:
# cat <<EOF>> reencrypt.py import os from subprocess import run import yaml fh=open("/etc/vmware/wcp/secrets-config.yaml") config=yaml.load(fh, Loader=yaml.FullLoader) for file_id, file_info in config["files"].items(): if isinstance(file_info, dict): path = file_info["path"] else: path = file_info if os.path.exists(path): run(["/usr/lib/vmware-wcp/hypercrypt.py", "--file-name", file_id, "--read", path]) EOF # Press Ctrl+D to finish editing reencrypt.py. -
Run reencrypt.py
python ./reencrypt.py
-
Additional Information
Troubleshooting Certmgr
- All logs for this tool are logged in the VCSA under /var/log/vmware/certmgr.log
- In case the 'certmgr' tool returns no output or fails with below error, ensure that you are running the command from /root/ of the vCenter Server.
"Error running supervisor cert manager: error while running CPVM cmd: Error running cmd on (##.##.##.##): /tmp/certmgr certificates list --json, error: Process exited with status 127"
Notes on certificates and keys that are not renewed by certmgr:
- Main vSphere with Tanzu Certificate KB: vSphere with Tanzu Certificate Guide
/etc/vmware/wcp/tls/authproxy-client.crtand/etc/vmware/wcp/tls/pinniped.crtcertificates on the Supervisor control plane VMs are not renewed by the certmgr script.- These certs are auto-regenerated when their expiration hits 2/3 of their overall lifetime.
- If these certificates have expired, open a ticket with VMware by Broadcom support for assistance in regenerating them.
- These two certificates cannot be replaced with custom certificates.
- Supervisor cluster system pod certificates are not managed by the certmgr script. These certificates are managed instead by the cert-manager system pod in the Supervisor cluster and are expected to automatically renew before expiry. If these certificates are found to be expired, see the following KB:
vSphere Supervisor System Pod Certificate Expiry due to Cert-Manager Issues - Older versions of certmgr tools do not rotate spherelet certificates, which will leave Supervisor Clusters worker nodes (ESXi hosts) in 'Not Ready' state and Host Config in Configuring state. It is important to use the latest certmgr tool from this KB.
Admin.Conf Certificates
- Note - Nov 14, 2025: The certmgr tool had an issue where it does not always successfully rotate the admin.conf 's certificates. This should be resolved in v0.0.19, but if the issue is observed, open a ticket with Broadcom Support to troubleshoot and fix the script.
- When these admin.conf certificates are expired, kubectl commands will fail.
- See the following KB article to manually rotate the admin.conf certificates:
Unable to run kubectl command on a Supervisor cluster CPVM with "error: You must be logged in to the server (Unauthorized)"
ESXi Host Not Ready - Spherelet Certificates
- If the ESXi hosts show Not Ready state after running the 'certmgr' tool, see ESXi nodes become NotReady after rotating Supervisor Certificates using certmgr
Kubelet.Conf Certificates
- Note - Nov 14,2025: The certmgr tool had an issue where it did not always successfully rotate the kubelet.conf 's embedded certificate. This should be resolved in v0.0.19, but if the issue is observed, open a ticket with Broadcom Support to troubleshoot and fix the script.
- See the following KB article to manually rotate the embedded certificate in kubelet.conf:
Supervisor controlPlane Node NotReady Error "unable to load bootstrap kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory"
Updates
- v0.0.19 - Nov 12,2025:
-
- Enforce TLS to use FIPS approved settings
- Added subtask to restart wcp-fip pod after the cert-rotation
- Handle apiserver-proxy server/client certs rotation and removes supervisor-nginx cert
- Added retry while failed to stop the container
- Improve list command to print client cert expiry from config files
- Add logic in spherelet cert generation code to take timezone difference in account
- Add steps to restart static pods after hypercrypt runs.
- Add appplatform authz cert rotation
- Added kubeconfig in kubectl commands
- Container restart for appplatform authorization webhook after cert rotation
- Add cert rotation for apiserver webhook cert
- Support cert rotation for mgmt-image-proxy.crt and supervisor-nginx.crt
- Added step to reencrypt keys using hypercrypt after rotation
Attachments
Feedback
Yes
No
Powered by
