Products

VMware vCenter Server VMware vSphere Kubernetes Service Tanzu Kubernetes Runtime

Issue/Introduction

vSphere with Tanzu / vSphere Kubernetes Service Supervisor Cluster certificates or ESXi spherelet certificates have expired or are about to expire.

The following commands can be used to check the expiration of Kubernetes certificates on the Supervisor cluster while using SSH to access each Supervisor control plane VM:

#Checks for wcp/tls and kubernetes/pki certificates
find /etc -type f \( -name "*.cert" -o -name "*.crt" \)  | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'

#Check for the kubelet certificate's expiration
openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet.crt | grep After

#Checks that the admin.conf file's certificates are not expired
cat /etc/kubernetes/admin.conf | grep certificate-authority-data | awk '{print $2}' | base64 -d | openssl x509 -noout -text | grep After

cat /etc/kubernetes/admin.conf | grep client-certificate-data | awk '{print $2}' | base64 -d | openssl x509 -noout -text | grep After

The attached certmgr script can be used to check for certificate expiration dates with the following commands when run from the VCSA:

#Retrieve the ID of the Supervisor Cluster to check its certificates
./certmgr supervisors

#Use certmgr to list the certificates for the Supervisor Cluster
./certmgr certificates list -c <domain id for the supervisor cluster>

Example output of the above command:

/root/certmgr certificates list
time="2025-12-03T09:27:59Z" level=info msg="[/root/certmgr certificates list]"
+--------------+--------------+----------------------------------+------------------------------------------------------------------+-------------------------------+-----------+
|    SCOPE     |      IP      |             HOSTNAME             |                               NAME                               |           NOTAFTER            | ISEXPIRED |
+--------------+--------------+----------------------------------+------------------------------------------------------------------+-------------------------------+-----------+
| controlplane | ##.###.##.## |    <Supervisor-Node-Name>        | /etc/kubernetes/admin.conf(client-certificate-data)              | 2026-12-03 07:08:42 +0000 UTC | false     |
|              |              |                                  | /etc/kubernetes/controller-manager.conf(client-certificate-data) | 2026-12-03 07:08:42 +0000 UTC | false     |
|              |              |                                  | /etc/kubernetes/pki/front-proxy-client.crt                       | 2026-12-03 07:08:42 +0000 UTC | false     |
|              |              |                                  | /etc/kubernetes/pki/apiserver.crt                                | 2026-12-03 07:08:42 +0000 UTC | false     |
|              |              |                                  | /etc/kubernetes/pki/apiserver-etcd-client.crt                    | 2026-12-03 07:08:42 +0000 UTC | false     |

Workload/Guest cluster certificates use the same certmgr script when run from a jumpbox with kubectl CLI or from the Supervisor cluster. The instructions can be found in the following KB:

Replace vSphere with Tanzu Guest Cluster Certificates

Cause

Kubernetes certificates are set to expire after a year by default. VMware by Broadcom Kubernetes products adhere to this certificate expiry timeline.

Resolution

NOTE: Before running the attached certmgr tool, check the below items to make sure of the following:

Time is synchronized across the environment, such as vCenter Server, ESXi, Supervisor Control Plane VMs.
Supervisor 'Config Status' is Running and 'Host Config Status' is Running. See Workload Management configuration monitoring
All nodes in Supervisor cluster are Ready. See Connect to the Supervisor Cluster as a vCenter Single Sign-On User

Install the wcp_cert_manager tool to vCenter

Move the attached file titled wcp_cert_manager.zip to the vCenter Server (under /root/) where vSphere with Tanzu is deployed. (Use WinSCP from Windows OS if required):
```
$ scp ./wcp_cert_manager.zip root@##.##.##.##:/root
```
If you encounter an issue using WinSCP, see the following KB article: Connecting to vCenter Server Virtual Appliance using WinSCP fails
Make sure you are under /root and unzip the file:

$ unzip wcp_cert_manager.zip
Archive:  wcp_cert_manager.zip
  inflating: certmgr

$ ls -l
total 30956
-rwxr-xr-x 1 root root 23019418 MM DD HH:MM certmgr
-rw-r--r-- 1 root root  8675846 MM DD HH:MM wcp_cert_manager.zip

Using the wcp_cert_manager tool

From /root run './certmgr certificates rotate' command to rotate all supervisor control plane certificates and spherelet certificates.

If you have multiple vSphere with Tanzu deployments on your vCenter, then you need to use the -c argument to specify the cluster you want to replace certificates on.
In order to gather the supervisor cluster id you can run:

$ ./certmgr supervisors
YYYY/MM/DD HH:MM:DD Cluster: domain-c#:########-####-####-####-#############

An example of running the tool on a specific cluster would be:
./certmgr certificates rotate -c domain-c#:########-####-####-####-#############

To ensure the rotated certificate keys are persisted upon reboot:
1. SSH into each Supervisor control plane VM:
  
  See SSH into Supervisor Control Plane VM Instructions: Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's (323407)
2. Perform the following script command on each Supervisor control plane VM:
```
/usr/lib/vmware-wcp/hypercrypt.py --reencrypt
```

NOTE: In a Legacy vSphere 7.0 environments, since hypercrypt.py lacks the reencrypt option, create reencrypt.py as a substitute and run that Python script:

Create reencrypt.py as follows:

# cat <<EOF>> reencrypt.py
import os
from subprocess import run
import yaml


fh=open("/etc/vmware/wcp/secrets-config.yaml")
config=yaml.load(fh, Loader=yaml.FullLoader)

for file_id, file_info in config["files"].items():
    if isinstance(file_info, dict):
        path = file_info["path"]
    else:
        path = file_info

    if os.path.exists(path):
        run(["/usr/lib/vmware-wcp/hypercrypt.py", "--file-name", file_id, "--read", path])
EOF

# Press Ctrl+D to finish editing reencrypt.py.

Run reencrypt.py
```
python ./reencrypt.py
```

`certmgr` Usage Guide

WCP Certificates Manager is a CLI tool for provisioning, viewing, and rotating certificates for VMware Cloud Platform (WCP) on vCenter or Supervisor Control Plane VMs.

Overview

The tool is designed to be run from vCenter, and it can automatically copy itself to control plane VMs to execute commands remotely when needed.

Global Flags

Flag	Short	Description
`--debug`	`-d`	Enable debug logging
`--leader`	`-l`	Toggle this command runner as a leader (only applies on control plane VMs)
`--json`	`-j`	Enable JSON output

Commands

`certmgr version`

Display version information.

certmgr version

`certmgr supervisors`

Lists all Supervisors in the system.

certmgr supervisors
certmgr supervisors list

Subcommands:

list – Lists all Supervisors in the system
copy [CLUSTER_ID] – Copies this program to all Supervisor control plane VMs. If a cluster ID is provided, only targets that cluster.

certmgr supervisors copy
certmgr supervisors copy domain-c123

`certmgr certificates`

Certificate management commands for Supervisors.

Flags (persistent):

Flag	Short	Description
`--cluster`	`-c`	vSphere cluster ID to target

Subcommands:

`list`

Lists all certificates.

certmgr certificates list
certmgr certificates list --json
certmgr certificates list -c domain-c123

`check-expiration`

Checks the expiration of all certificates.

certmgr certificates check-expiration
certmgr certificates check-expiration --days 30
certmgr certificates check-expiration --json

Flag	Short	Default	Description
`--days`	`-e`	30	Duration in days from now to consider certificates in need of rotation
`--json`		false	Output in JSON format

`backup`

Backup all certificates to the specified directory.

certmgr certificates backup
certmgr certificates backup --backup-dir /custom/backup/path

Flag	Default	Description
`--backup-dir`	`/etc/kubernetes/pki.bak`	Directory to use for certificate backups (must exist)

`rotate`

Rotates all certificates.

certmgr certificates rotate
certmgr certificates rotate --backup-dir /custom/backup/path
certmgr certificates rotate --spherelet-only
certmgr certificates rotate --vmca

Flag	Short	Default	Description
`--backup-dir`		`/etc/kubernetes/pki.bak`	Directory for certificate backups
`--vmca`	`-v`	false	[Experimental] Rotates VMCA-signed certificates
`--spherelet-only`	`-s`	false	Only rotate spherelet certificates
`--json`		false	Output in JSON format

`certmgr tkc`

Commands for Tanzu Kubernetes Clusters (guest clusters).

Flags (persistent):

Flag	Short	Description
`--namespace`	`-n`	Namespace to use for cluster operations (defaults to current context namespace)

Subcommands:

`list`

Lists clusters in the system.

certmgr tkc list
certmgr tkc list -n my-namespace
certmgr tkc list --all-namespaces

Flag	Short	Description
`--all-namespaces`	`-a`	List clusters across all namespaces

`certificates list NAME`

List control plane certificates for a guest cluster.

certmgr tkc certificates list my-cluster
certmgr tkc certificates list my-cluster -n my-namespace

`certificates rotate NAME`

Rotate control plane certificates for a guest cluster.

certmgr tkc certificates rotate my-cluster
certmgr tkc certificates rotate my-cluster -n my-namespace

Example Workflows

Check for expiring certificates (within 60 days)

certmgr certificates check-expiration --days 60

Backup and rotate all certificates

certmgr certificates backup --backup-dir /tmp/cert-backup
certmgr certificates rotate --backup-dir /tmp/cert-backup

Rotate certificates for a specific cluster

certmgr certificates rotate -c domain-c123

List and rotate TKC certificates

certmgr tkc list --all-namespaces
certmgr tkc certificates list my-tkc-cluster -n my-namespace
certmgr tkc certificates rotate my-tkc-cluster -n my-namespace

Additional Information

Troubleshooting Certmgr

All logs for this tool are logged in the VCSA under /var/log/vmware/certmgr.log
In case the 'certmgr' tool returns no output or fails with below error, ensure that you are running the command from /root/ of the vCenter Server.

"Error running supervisor cert manager: error while running CPVM cmd: Error running cmd on (##.##.##.##): /tmp/certmgr certificates list --json, error: Process exited with status 127"

Notes on certificates and keys that are not renewed by certmgr:

Main vSphere with Tanzu Certificate KB: vSphere with Tanzu Certificate Guide
/etc/vmware/wcp/tls/authproxy-client.crt and /etc/vmware/wcp/tls/pinniped.crt certificates on the Supervisor control plane VMs are not renewed by the certmgr script.
- These certs are auto-regenerated when their expiration hits 2/3 of their overall lifetime.
- If these certificates have expired, open a ticket with VMware by Broadcom support for assistance in regenerating them.
- These two certificates cannot be replaced with custom certificates.
Supervisor cluster system pod certificates are not managed by the certmgr script. These certificates are managed instead by the cert-manager system pod in the Supervisor cluster and are expected to automatically renew before expiry. If these certificates are found to be expired, see the following KB:

vSphere Supervisor System Pod Certificate Expiry due to Cert-Manager Issues
Older versions of certmgr tools do not rotate spherelet certificates, which will leave Supervisor Clusters worker nodes (ESXi hosts) in 'Not Ready' state and Host Config in Configuring state. It is important to use the latest certmgr tool from this KB.

Admin.Conf Certificates

Note - Nov 14, 2025: The certmgr tool had an issue where it does not always successfully rotate the admin.conf 's certificates. This should be resolved in v0.0.19, but if the issue is observed, open a ticket with Broadcom Support to troubleshoot and fix the script.
When these admin.conf certificates are expired, kubectl commands will fail.
See the following KB article to manually rotate the admin.conf certificates:

Unable to run kubectl command on a Supervisor cluster CPVM with "error: You must be logged in to the server (Unauthorized)"

ESXi Host Not Ready - Spherelet Certificates

If the ESXi hosts show Not Ready state after running the 'certmgr' tool, see ESXi nodes become NotReady after rotating Supervisor Certificates using certmgr

Kubelet.Conf Certificates

Note - Nov 14,2025: The certmgr tool had an issue where it did not always successfully rotate the kubelet.conf 's embedded certificate. This should be addressed in the future version; if this issue is observed open a ticket with Broadcom Support to troubleshoot and fix the script.
See the following KB article to manually rotate the embedded certificate in kubelet.conf:

Supervisor controlPlane Node NotReady Error "unable to load bootstrap kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory"

Updates

v0.0.19 - Nov 12,2025:

- Enforce TLS to use FIPS approved settings
- Added subtask to restart wcp-fip pod after the cert-rotation
- Handle apiserver-proxy server/client certs rotation and removes supervisor-nginx cert
- Added retry while failed to stop the container
- Improve list command to print client cert expiry from config files
- Add logic in spherelet cert generation code to take timezone difference in account
- Add steps to restart static pods after hypercrypt runs.
- Add appplatform authz cert rotation
- Added kubeconfig in kubectl commands
- Container restart for appplatform authorization webhook after cert rotation
- Add cert rotation for apiserver webhook cert
- Support cert rotation for mgmt-image-proxy.crt and supervisor-nginx.crt
- Added step to reencrypt keys using hypercrypt after rotation

Attachments

wcp_cert_manager_v0.0.20.zip get_app

Replace vSphere with Tanzu / vSphere Kubernetes Service Supervisor Certificates

Article ID: 322994

Updated On:

Products

Issue/Introduction

Cause

Resolution

NOTE: Before running the attached certmgr tool, check the below items to make sure of the following:

Install the wcp_cert_manager tool to vCenter

Using the wcp_cert_manager tool

certmgr Usage Guide

Overview

Global Flags

Commands

certmgr version

certmgr supervisors

Subcommands:

certmgr certificates

Flags (persistent):

Subcommands:

list

check-expiration

backup

rotate

certmgr tkc

Flags (persistent):

Subcommands:

list

certificates list NAME

certificates rotate NAME

Example Workflows

Check for expiring certificates (within 60 days)

Backup and rotate all certificates

Rotate certificates for a specific cluster

List and rotate TKC certificates

Additional Information

Troubleshooting Certmgr

Notes on certificates and keys that are not renewed by certmgr:

Admin.Conf Certificates

ESXi Host Not Ready - Spherelet Certificates

Kubelet.Conf Certificates

Updates

Attachments

Feedback

`certmgr` Usage Guide

`certmgr version`

`certmgr supervisors`

`certmgr certificates`

`list`

`check-expiration`

`backup`

`rotate`

`certmgr tkc`

`list`

`certificates list NAME`

`certificates rotate NAME`