vSphere Supervisor Certificates - authproxy-client.crt, pinniped.crt, vip.crt, wcp-node-bootstrapper certificates - "tls: failed to verify certificate: x509: certificate has expired or is not yet valid"
Article ID: 389860
Updated On:
Products
Issue/Introduction
This KB is written for advising on the following vSphere Supervisor certificates which are not included in the renewal performed by the certmgr script:
- /etc/vmware/wcp/tls/authproxy-client.crt
- /etc/vmware/wcp/tls/pinniped.crt
- /etc/vmware/wcp/tls/vip.crt
- /var/etcdmgr/signer/crt/wcp_node_bootstrapper
Certificates that are renewed by the certmgr script can be found by running the following certmgr script command while SSH into the VCSA:
- Retrieve the Supervisor cluster ID:
./certmgr supervisors
- View all certificates on the chosen Supervisor cluster ID, replacing "domain-c#:########-####-####-####-#############" with the desired Supervisor cluster's ID from above:
./certmgr certificates list -c domain-c#:########-####-####-####-#############
A list of vSphere Supervisor certificates can be found while directly SSH into a Supervisor control plane VM and performing the following find command:
find / -type f \( -name "*.cert" -o -name "*.crt" \) -print 2>/dev/null | egrep -iv 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd|backup' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'
May also notice the below error on masterproxy-tkgs-plugin:
Error creating: Internal error occurred: failed calling webhook "admission.vmware.com": failed to call webhook: Post "https://127.0.0.1:12345/scheduler/admission?timeout=10s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time ####### is after ######.
Environment
vSphere with Tanzu 7.0
vSphere with Tanzu 8.0
This issue can occur on a vSphere Kubernetes cluster regardless of whether or not it is managed by Tanzu Mission Control (TMC)
Cause
vSphere Supervisor Certificates can be renewed through the certmgr script provided in KB: Replace vSphere with Tanzu Supervisor Certificates
However, this certmgr script does not renew the authproxy-client, pinniped, vip and wcp_node_bootstrapper certificates:
- The /etc/vmware/wcp/tls/vip.crt also known as the Workload Platform Management certificate can be renewed through the steps in KB: Replace the Self-Signed vip.crt
- /etc/vmware/wcp/tls/authproxy-client.crt and /etc/vmware/wcp/tls/pinniped.crt are managed and automatically renewed by the cert-manager controller in the Supervisor cluster.
- It is currently not supported to use custom certificates for these certificates.
- It is currently not supported to use custom certificates for these certificates.
- /var/etcdmgr/signer/crt/wcp_node_bootstrapper is only used on initial creation of a Supervisor cluster and does not need to be renewed.
All certificates are automatically renewed upon successful Supervisor cluster upgrade.
Resolution
A list of vSphere Supervisor certificates can be found while directly SSH into a Supervisor control plane VM and performing the following find command:
find / -type f \( -name "*.cert" -o -name "*.crt" \) -print 2>/dev/null | egrep -iv 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd|backup' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'
Supervisor Certificates can be renewed through the certmgr script provided in the following KB:
- Replace vSphere with Tanzu Supervisor Certificates
- However, this certmgr script does not renew the authproxy-client, pinniped, vip and wcp_node_bootstrapper certificates.
authproxy-client.crt and pinniped.crt
- The /etc/vmware/wcp/tls/authproxy-client.crt is an internally generated client certificate which allows for nginx to make successful requests to authproxy.
- The /etc/vmware/wcp/tls/pinniped.crt is an internally generated CA certificate used for establishing trust between nginx and pinniped.
- These certificates are managed by the cert-manager controller in the Supervisor cluster and are expected to be automatically rotated before expiration by the cert-manager controller accordingly.
- Supervisor OIDC Federation requires both certificates.
- If Pinniped is not used in the environment, these certificates are also not used but the certificates will still be generated and renewed regardless.
- It is currently not supported to use custom certificates for these certificates.
If these certificates are expired, you can rotate them manually with the steps below.
# replace the authproxy-client certs
kubectl -n kube-system get secret wcp-authproxy-client-secret -o jsonpath='{.data.tls\.crt}' | base64 -d > /etc/vmware/wcp/tls/authproxy-client.crt
kubectl -n kube-system get secret wcp-authproxy-client-secret -o jsonpath='{.data.tls\.key}' | base64 -d > /etc/vmware/wcp/tls/authproxy-client.key
kubectl -n kube-system get secret wcp-authproxy-client-secret -o jsonpath='{.data.ca\.crt}' | base64 -d > /etc/vmware/wcp/tls/authproxy-client-ca.crt
# replace the pinniped cert
kubectl -n vmware-system-pinniped get secret pinniped-supervisor-default-tls-certificate -o jsonpath='{.data.ca\.crt}' | base64 -d > /etc/vmware/wcp/tls/pinniped.crt
# then restart the kubectl-plugin-vsphere on *each* supervisor node
root@supervisor1 [ ~ ]# crictl ps | grep kubectl-plugin-vsphere
XX YY 6 days ago Running kubectl-plugin-vsphere 4 AA kubectl-plugin-vsphere-supervisor1
root@supervisor1 [ ~ ]# crictl stop XX
ZZ
# check the pods are running
root@supervisor1 [ ~ ]# kubectl get pods -n kube-system | grep kubectl-plugin-vsphere
kubectl-plugin-vsphere-supervisor1 1/1 Running 5 (6d ago) 6d
kubectl-plugin-vsphere-supervisor2 1/1 Running 5 (6d ago) 6d
kubectl-plugin-vsphere-supervisor3 1/1 Running 5 (20s ago) 6dvip.crt - Workload Platform Management
- The /etc/vmware/wcp/tls/vip.crt also known as the Workload Platform Management certificate can be renewed through the steps in KB: Replace the Self-Signed vip.crt
wcp_node_bootstrapper
- /var/etcdmgr/signer/crt/wcp_node_bootstrapper is only used on initial creation of a Supervisor cluster and does not need to be renewed.
Feedback
