PVC or VolumeSnapshot Operations Fail on Supervisor Cluster Due to Expired Storage Quota Certificates

Products

• VMware vCenter Server

Issue/Introduction

On a Supervisor cluster, creating or updating PVC or VolumeSnapshot objects may occasionally fail during storage quota validation. This issue manifests when the storage quota webhook and CNS extension service use expired certificate data.

You may observe an error message similar to one of the following error messages in your environment:

failed to create volume : admission webhook "validate-quota-on-create.k8s.io" denied the request: Operation denied, Post "https://cns-vsphere-vmware-com-service.kube-system.svc.cluster.local:443/getrequestedcapacityforpersistentvolumeclaim": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-08-27T16:58:17Z is after 2025-08-23T02:15:00Z

OR

failed to create volume : admission webhook "validate-quota-on-create.k8s.io" denied the request: Operation denied, Post "https://cns-vsphere-vmware-com-service.kube-system.svc.cluster.local:443/getrequestedcapacityforpersistentvolumeclaim": remote error: tls: expired certificate

Environment

vCenter Server: 9.0, 9.0.1, 9.0.2

Cause

During storage quota validation, the storage quota webhook communicates with the CNS extension service via an mTLS connection. This connection relies on client-server certificates managed by cert-manager.

While cert-manager successfully auto-renews these certificates upon expiry, the new certificate data is not automatically reloaded into the storage quota webhook or CNS extension service pods. Consequently, the services continue to use old, expired certificate data, causing connection failures.

Resolution

Resolution

To resolve this issue, you must restart the storage quota webhook and CNS extension service pods to force them to reload the new certificate data.

 

1. SSH into the vCenter appliance as root: ssh root@<vcenter-ip>

2. Retrieve the credentials for the Supervisor control plane: /usr/lib/vmware-wcp/decryptK8Pwd.py

3. SSH into the Supervisor control plane using the IP and credentials obtained in the previous step: ssh root@<supervisor-ip>

4. Scale down the storage-quota-webhook pods to zero: kubectl -n kube-system scale deploy storage-quota-webhook --replicas=0

5. Scale the storage-quota-webhook pods back up (the default is typically 3): kubectl -n kube-system scale deploy storage-quota-webhook --replicas=3

6. Scale down the cns-storage-quota-extension pods to zero: kubectl -n kube-system scale deploy cns-storage-quota-extension --replicas=0

7. Scale the cns-storage-quota-extension pods back up (the default is 1): kubectl -n kube-system scale deploy cns-storage-quota-extension --replicas=1