Unable to create new PVCs two months after supervisor creation

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• After the vSphere Kubernetes Supervisor service is deployed, you're able to create Persistent Volume Claims (PVCs), but after two months PVCs fail to create.
  
  
• From the supervisor context, running kubectl logs -f -n vmware-system-csi -l app=vsphere-csi-controller displays the error:

<TIMESTAMP> stderr F {"level":"error","time":"<TIMESTAMP>","caller":"wcpguest/controller.go:358","msg":"failed to create pvc with name: <UUID> on namespace: <NAMESPACE> in supervisorCluster. Error: admission webhook \"validate-quota-on-create.k8s.io\" denied the request: Operation denied, Post \"https://cns-vsphere-vmware-com-service.kube-system.svc.cluster.local:443/getrequestedcapacityforpersistentvolumeclaim\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time <TIMESTAMP> is after<TIMESTAMP>","TraceId":"<UUID>","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/wcpguest.(*controller).CreateVolume.func1\n

Environment

vSphere Kubernetes Service 9.0.0

Cause

The certificates for the storage-quota-webhook and cns-storage-quota-extension deployments expire after 2 months. Cert manager generates new certificates for both of these deployments, but their pods don't restart to pick up the new certificate.

Resolution

From the supervisor context, run the following:

• kubectl -n kube-system rollout restart deploy storage-quota-webhook
• kubectl -n kube-system rollout restart deploy cns-storage-quota-extension

Additional Information

This is a known issue that will be resolved in a future release.

• CA cert in Secret not updated when self-signed CA itself gets renewed.
• Certmgr allows creating certificates expiring after ca expiration.