How to rotate the certification - vmware-system-csi-webhook-service

Products

• VMware vSphere ESXi
• VMware vSphere Kubernetes Service

Issue/Introduction

Provide the procedure for how to rotate the certification of vmware-system-csi-webhook-service when facing "x509: certificate has expired".

Symptoms:
The new pod with PV creation failed due to expired certification of vmware-system-csi-webhook-service in vSphere with Tanzu.

kubectl describe pvc $TARGET_PVC

Warning ProvisioningFailed 57s csi.vsphere.vmware.com_vsphere-csi-controller-5cff94c9f6-hdcg4_acfbe180-1e85-47c7-bcc7-08f2d220cf1b failed to provision volume with StorageClass "tanzu-storage": rpc error: code = Internal desc = failed to create pvc with name: f7024439-7ec3-4400-a4c2-c033896e484e-86c16ce2-1c51-4eb0-a6d0-1ba543bba5f1 on namespace: XXXX in supervisorCluster. Error: Internal error occurred: failed calling webhook "validation.csi.vsphere.vmware.com": failed to call webhook: Post "https://vmware-system-csi-webhook-service.vmware-system-csi.svc:443/validate?timeout=10s": x509: certificate has expired or is not yet valid: current time 2023-11-22T08:38:00Z is after 2023-08-23T09:14:02Z

You can check the target certification expired date.

# Run as Supervisor context
kubectl get secret -n vmware-system-csi vmware-system-csi-webhook-service-cert -o jsonpath='{.data.ca\.crt}'|base64 -d | openssl x509 -noout -dates
#> notBefore=May 25 09:14:02 2023 GMT
#> notAfter=Aug 23 09:14:02 2023 GMT

Environment

VMware vSphere 7.0 with Tanzu

Resolution

Workaround:
Notice: This procedure is verified in vSphere 8, not vSphere 7.

Rotate vmware-system-csi-webhook-service-cert forcibly in the Supervisor node.

# Jumpbox
ssh root@${vcenter}

# Get password of Supervisor
/usr/lib/vmware-wcp/decryptK8Pwd.py

# SSH to supervisor
ssh root@${supervisor_ipadderss}

# Check the target of secret 
kubectl get secret -n vmware-system-csi vmware-system-csi-webhook-service-cert -o jsonpath='{.data.ca\.crt}'|base64 -d | openssl x509 -noout -dates
#> notBefore=May 25 09:14:02 2023 GMT
#> notAfter=Aug 23 09:14:02 2023 GMT

# Delete the secret 
kubectl delete secret -n vmware-system-csi vmware-system-csi-webhook-service-cert

# New secret will be generated
kubectl get secret -n vmware-system-csi vmware-system-csi-webhook-service-cert

# Check the cert expired date is updated
kubectl get secret -n vmware-system-csi vmware-system-csi-webhook-service-cert -o jsonpath='{.data.ca\.crt}'|base64 -d | openssl x509 -noout -dates