Replace vSphere with Tanzu Supervisor Certificates
Article ID: 322994
Updated On:
Products
VMware vCenter Server
VMware vSphere Kubernetes Service
Tanzu Kubernetes Runtime
Issue/Introduction
vSphere with Tanzu Supervisor Cluster certificates or ESXi spherelet certificates have expired or are about to expire.
The following commands can be used to check the expiration of Kubernetes certificates on the Supervisor cluster:
While SSH to each Supervisor control plane VM:
#Checks for wcp/tls and kubernetes/pki certificates
find /etc -type f \( -name "*.cert" -o -name "*.crt" \) | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'
#Check for the kubelet certificate's expiration
openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet.crt | grep After
#Checks that the admin.conf file's certificates are not expired
cat /etc/kubernetes/admin.conf | grep certificate-authority-data | awk '{print $2}' | base64 -d | openssl x509 -noout -text | grep After
cat /etc/kubernetes/admin.conf | grep client-certificate-data | awk '{print $2}' | base64 -d | openssl x509 -noout -text | grep After
The attached certmgr script can be used to check for certificate expiration dates with the following commands when run from the VCSA:
#Retrieve the ID of the Supervisor Cluster to check its certificates
./certmgr supervisors
#Use certmgr to list the certificates for the Supervisor Cluster
./certmgr certificates list -c <domain id for the supervisor cluster>
Workload/Guest cluster certificates use the same certmgr script when run from a jumpbox with kubectl CLI or from the Supervisor cluster.
The instructions can be found in the following KB:
Environment
vSphere Supervisor 7
vSphere Supervisor 8
vSphere Supervsior 9
Cause
Kubernetes certificates are set to expire after a year by default. VMware by Broadcom Kubernetes products adhere to this certificate expiry timeline.
Resolution
NOTE: Before running the attached certmgr tool, check the below items to make sure of the following:
- Time synchronized across the environment, such as vCenter Server, ESXi, Supervisor Control Plane VMs.
- Supervisor '
Config Status' isRunningand 'Host Config Status' isRunning. See Workload Management configuration monitoring - All nodes in Supervisor cluster are
Ready. See Connect to the Supervisor Cluster as a vCenter Single Sign-On User
Install the wcp_cert_manager tool to vCenter
- Move the attached file titled wcp_cert_manager.zip to the vCenter Server (under /root/ ) where vSphere with Tanzu is deployed. (Use WinSCP from Windows OS if required):
$ scp ./wcp_cert_manager.zip root@##.##.##.##:/root The authenticity of host '##.##.##.## (##.##.##.##)' can't be established. ECDSA key fingerprint is SHA256:<fingerprint>. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '##.##.##.##' (ECDSA) to the list of known hosts. VMware vCenter Server #.0.#.##### Type: vCenter Server with an embedded Platform Services Controller Password: wcp_cert_manager.zip 100% 8473KB 8.3MB/s 00:00If you encounter an issue using WinSCP, please see the following KB article:
Connecting to vCenter Server Virtual Appliance using WinSCP fails - Make sure you are under /root and unzip the file:
pwd /root unzip wcp_cert_manager.zip Archive: wcp_cert_manager.zip inflating: certmgr ls -l total 30956 -rwxr-xr-x 1 root root 23019418 MM DD HH:MM certmgr -rw-r--r-- 1 root root 8675846 MM DD HH:MM wcp_cert_manager.zip
Using the wcp_cert_manager tool
- From /root run './certmgr certificates rotate' command to rotate all supervisor control plane certificates and spherelet certificates.
Below is an example of a successful cert rotation:./certmgr certificates rotate +------------------+------------------------------------------------------------------------------------------------------+-------+ | CONTROL PLANE IP | RESULT | ERROR | +------------------+------------------------------------------------------------------------------------------------------+-------+ | ##.##.##.## | +---------------------------------------------------------------------------------+----------------+ | | | | | TASKS | OVERALL STATUS | | | | | +---------------------------------------------------------------------------------+----------------+ | | | | | +--------------------------------+------------------------------------+-------+ | ok | | | | | | | TASK | RESULT | ERROR | | | | | | | | +--------------------------------+------------------------------------+-------+ | | | | | | | | backup certificates | /root/backups-16739895901776834456 | | | | | | | | | | rotate etcd server certificate | | | | | | | | | | | rotate api server etcd client | | | | | | | | | | | certificate | | | | | | | | | | | rotate etcd peer certificate | | | | | | | | | | | rotate etcd health check | | | | | | | | | | | certificate | | | | | | | | | | | rotate api server certificate | | | | | | | | | | | rotate kubelet client api | | | | | | | | | | | server certificate | | | | | | | | | | | rotate front proxy certificate | | | | | | | | | | | rotate controller-manager | | | | | | | | | | | certificate | | | | | | | | | | | rotate scheduler certificate | | | | | | | | | | | rotate scheduler extension | | | | | | | | | | | certificate | | | | | | | | | | | rotate kubelet certificate | | | | | | | | | | | restart ncp | NCP restart skipped: VDS setup | | | | | | | | | | | detected. | | | | | | | | | | rotate auth proxy certificate | | | | | | | | | | | rotate management certificate | | | | | | | | | | | rotate registry certificate | | | | | | | | | | | rotate kubeadm admin | | | | | | | | | | | certificate | | | | | | | | | | | verify etcd health | | | | | | | | | | +--------------------------------+------------------------------------+-------+ | | | | | | | | | | | | | +---------------------------------------------------------------------------------+----------------+ | | | | | | | ##.##.##.## | +---------------------------------------------------------------------------------+----------------+ | | | | | TASKS | OVERALL STATUS | | | | | +---------------------------------------------------------------------------------+----------------+ | | | | | +--------------------------------+------------------------------------+-------+ | ok | | | | | | | TASK | RESULT | ERROR | | | | | | | | +--------------------------------+------------------------------------+-------+ | | | | | | | | backup certificates | /root/backups-16739895893751688144 | | | | | | | | | | rotate etcd server certificate | | | | | | | | | | | rotate api server etcd client | | | | | | | | | | | certificate | | | | | | | | | | | rotate etcd peer certificate | | | | | | | | | | | rotate etcd health check | | | | | | | | | | | certificate | | | | | | | | | | | rotate api server certificate | | | | | | | | | | | rotate kubelet client api | | | | | | | | | | | server certificate | | | | | | | | | | | rotate front proxy certificate | | | | | | | | | | | rotate controller-manager | | | | | | | | | | | certificate | | | | | | | | | | | rotate scheduler certificate | | | | | | | | | | | rotate scheduler extension | | | | | | | | | | | certificate | | | | | | | | | | | rotate kubelet certificate | | | | | | | | | | | restart ncp | NCP restart skipped: VDS setup | | | | | | | | | | | detected. | | | | | | | | | | rotate auth proxy certificate | | | | | | | | | | | rotate management certificate | | | | | | | | | | | rotate registry certificate | | | | | | | | | | | rotate kubeadm admin | | | | | | | | | | | certificate | | | | | | | | | | | verify etcd health | | | | | | | | | | +--------------------------------+------------------------------------+-------+ | | | | | | | | | | | | | +---------------------------------------------------------------------------------+----------------+ | | | | | | | ##.##.##.## | +--------------------------------------------------------------------------------+----------------+ | | | | | TASKS | OVERALL STATUS | | | | | +--------------------------------------------------------------------------------+----------------+ | | | | | +--------------------------------+-----------------------------------+-------+ | ok | | | | | | | TASK | RESULT | ERROR | | | | | | | | +--------------------------------+-----------------------------------+-------+ | | | | | | | | backup certificates | /root/backups-1673989589793637456 | | | | | | | | | | rotate etcd server certificate | | | | | | | | | | | rotate api server etcd client | | | | | | | | | | | certificate | | | | | | | | | | | rotate etcd peer certificate | | | | | | | | | | | rotate etcd health check | | | | | | | | | | | certificate | | | | | | | | | | | rotate api server certificate | | | | | | | | | | | rotate kubelet client api | | | | | | | | | | | server certificate | | | | | | | | | | | rotate front proxy certificate | | | | | | | | | | | rotate controller-manager | | | | | | | | | | | certificate | | | | | | | | | | | rotate scheduler certificate | | | | | | | | | | | rotate scheduler extension | | | | | | | | | | | certificate | | | | | | | | | | | rotate kubelet certificate | | | | | | | | | | | restart ncp | NCP restart skipped: NCP | | | | | | | | | | | restart only occurs on the | | | | | | | | | | | leader. | | | | | | | | | | rotate auth proxy certificate | | | | | | | | | | | rotate management certificate | | | | | | | | | | | rotate registry certificate | | | | | | | | | | | rotate kubeadm admin | | | | | | | | | | | certificate | | | | | | | | | | | verify etcd health | | | | | | | | | | +--------------------------------+-----------------------------------+-------+ | | | | | | | | | | | | | +--------------------------------------------------------------------------------+----------------+ | | | | | | +------------------+------------------------------------------------------------------------------------------------------+-------+ +-----------------------------------------------------+----------------+ | TASKS | OVERALL STATUS | +-----------------------------------------------------+----------------+ | +--------------------------------+--------+-------+ | | | | TASK | RESULT | ERROR | | | | +--------------------------------+--------+-------+ | | | | rotate spherelet certificates | | | | | | | on ##.##.##.## (host-##) | | | | | | | rotate spherelet certificates | | | | | | | on ##.##.##.## (host-##) | | | | | | | rotate spherelet certificates | | | | | | | on ##.##.##.## (host-##) | | | | | | +--------------------------------+--------+-------+ | | | | | +-----------------------------------------------------+----------------+
If you have multiple vSphere with Tanzu deployments on your vCenter, then you need to use the -c argument to specify the cluster you want to replace certificates on.
In order to gather the supervisor cluster id you can run:
./certmgr supervisors
./certmgr supervisors
YYYY/MM/DD HH:MM:DD Cluster: domain-c#:########-####-####-####-#############
IP: ##.##.##.##
Password: ***********************************
In the above example the cluster id would be "domain-c#:########-####-####-####-#############"
An example of running the tool on a specific cluster would be:
./certmgr certificates list -c domain-c#:########-####-####-####-#############
./certmgr certificates rotate -c domain-c#:########-####-####-####-#############
To ensure the rotated certificate keys are persisted upon reboot:
- SSH into each Supervisor control plane VM:
See SSH into Supervisor Control Plane VM Instructions: Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's (323407) - Perform the following script command on each Supervisor control plane VM:
/usr/lib/vmware-wcp/hypercrypt.py --reencrypt
Additional Information
Troubleshooting Certmgr
All logs for this tool are logged in the VCSA under /var/log/vmware/certmgr.log
In case the 'certmgr' tool returns no output or fails with below error, ensure that you are running the command from /root/ of the vCenter Server.
"Error running supervisor cert manager: error while running CPVM cmd: Error running cmd on (##.##.##.##): /tmp/certmgr certificates list --json, error: Process exited with status 127"
Admin.Conf Certificates
Note - July 11, 2025: The certmgr tool currently has a bug where it does not always successfully rotate the admin.conf 's certificates.
When these admin.conf certificates are expired, kubectl commands will fail.
See the following KB article to manually rotate the admin.conf certificates:
Unable to run kubectl command on a Supervisor cluster CPVM with "error: You must be logged in to the server (Unauthorized)"
ESXi Host Not Ready - Spherelet Certificates
If the ESXi hosts show Not Ready state after running the 'certmgr' tool, see ESXi nodes become NotReady after rotating Supervisor Certificates using certmgr
Notes on certificates and keys that are not renewed by certmgr:
- Main vSphere with Tanzu Certificate KB: vSphere with Tanzu Certificate Guide
- /etc/vmware/wcp/tls/authproxy-client.crt and /etc/vmware/wcp/tls/pinniped.crt certificates on the Supervisor control plane VMs are not renewed by the certmgr script.
- These certs are auto-regenerated when their expiration hits 2/3 of their overall lifetime.
- If these certificates have expired, please open a ticket with VMware by Broadcom support for assistance in regenerating them.
- These two certificates cannot be replaced with custom certificates.
- Supervisor cluster system pod certificates are not managed by the certmgr script. These certificates are managed instead by the cert-manager system pod in the Supervisor cluster and are expected to automatically renew before expiry. If these certificates are found to be expired, see the following KB:
vSphere Supervisor System Pod Certificate Expiry due to Cert-Manager Issues - The rotation of root certificates kube-ca, etcd-ca, front-proxy-ca and the re-issuing of private keys are not supported by this certmgr tool. If you are in this situation, please refer to the following KB for resolution: Supervisor Cluster Unhealthy with etcd and kube-apiserver containers failing to start with error message "etcdmain: tls: private key does not match public key"
- Older versions of certmgr tools do not rotate spherelet certificates, which will leave Supervisor Clusters worker nodes (ESXi hosts) in 'Not Ready' state and Host Config in Configuring state. It is important to use the latest certmgr tool from this KB.
Attachments
Feedback
Yes
No
Powered by
