Prometheus deployment fails with PVC timeout on VKS Guest Cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When you attempt to deploy the Prometheus package on a VKS Guest cluster, the installation remains stuck at 31/32 changes. You observe that the prometheus-server pod stays in a Pending state with the following error messages:

| 3:50:34PM: ---- waiting on 1 changes [31/32 done] ----
            | 3:51:01PM: ongoing: reconcile deployment/prometheus-server (apps/v1) namespace: tanzu-system-monitoring
            | 3:51:01PM:  ^ Waiting for 1 unavailable replicas
          | 3:51:01PM:  L ok: waiting on replicaset/prometheus-server-#### (apps/v1) namespace: tanzu-system-monitoring
          | 3:51:01PM:  L ongoing: waiting on pod/prometheus-server-####-####(v1) namespace: tanzu-system-monitoring
            | 3:51:01PM:     ^ Pending: Unschedulable, message: 0/19 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found

• Describe output for the prometheus-server pod

kubectl describe pvc -n tanzu-system-monitoring prometheus-server

Used By:       prometheus-server-####-##
Events:

  Type     Reason                Age                  From                                                                                                Message

  ----     ------                ----                 ----                                                                                                -------

  Warning  ProvisioningFailed    16m                  csi.vsphere.vmware.com_vsphere-csi-controller-###  failed to provision volume with StorageClass "<guest-cluster-storageclass-name>": rpc error: code = Internal desc = failed to create volume on namespace: <Namespace-Name> in supervisor cluster. Error: persistentVolumeClaim #### in namespace <Namespace-Name> not in phase Bound within 240 seconds

  Warning  ProvisioningFailed    10m                  persistentvolume-controller                                                                         Error saving claim: Operation cannot be fulfilled on persistentvolumeclaims "prometheus-server": the object has been modified; please apply your changes to the latest version and try again

  Normal   Provisioning          16s (x6 over 20m)    csi.vsphere.vmware.com_vsphere-csi-controller-###  External provisioner is provisioning volume for claim "tanzu-system-monitoring/prometheus-server"

  Warning  ProvisioningFailed    16s (x4 over 12m)    csi.vsphere.vmware.com_vsphere-csi-controller-###  failed to provision volume with StorageClass "<guest-cluster-storageclass-name>": rpc error: code = Internal desc = failed to create volume on namespace: <Namespace-Name> in supervisor cluster. Error: persistentVolumeClaim #### in namespace <Namespace-Name> not in phase Bound within 240 seconds. reason: failed to provision volume with StorageClass "<guest-cluster-storageclass-name>": rpc error: code = DeadlineExceeded desc = context deadline exceeded

  Normal   ExternalProvisioning  10s (x100 over 20m)  persistentvolume-controller                                                                         Waiting for a volume to be created either by the external provisioner 'csi.vsphere.vmware.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

• Validate the ESXi host for any underlying kernel or storage related latency issue, as First Class Disk (FCD) allocation/creation process takes more time for CSI provisioner is configured to wait.

vmkwarning: cpu65:2099121)WARNING: ScsiDeviceIO: 1799: Device naa.### performance has deteriorated. I/O latency increased from average value of 20413 microseconds to 447471 microseconds.

Environment

VCF 9.x 
vSphere Kubernetes Service 

Cause

The issue occurs because the creation of the 150GB Persistent Volume (PV) on the underlying VMFS storage exceeds the default 4-minute (240-second) timeout.  

Resolution

To resolve this issue, you can increase the wait timeout for the package installation to allow sufficient time for the storage backend to provision the volume.

1. Increase the --wait-timeout to 6 minutes (6m0s) or higher when running the installation command: 

   • vcf package install prometheus -p prometheus.kubernetes.vmware.com -v 3.5.0+vmware.1-vks.2 --values-file prometheus-data-values.yaml -n vmware-system-tkg --wait-timeout 6m0s

2. If the volume provisioning continues to time out, perform a rollout restart of the vSphere CSI controller on both the Guest and Supervisor clusters to refresh the storage communication path:

   • Guest Cluster: 

     • kubectl rollout restart deployment/vsphere-csi-controller -n kube-system

   • Supervisor Cluster: 

     • kubectl rollout restart deployment/vsphere-csi-controller -n vmware-system-csi