vSphere with Tanzu PVC creation failure due to "No candidates datastores found"

Products

• VMware vSphere ESXi
• VMware vSphere with Tanzu

Issue/Introduction

Detail a known condition for more rapid resolution.

Symptoms:

• vSphere with Tanzu fails to provision PersistentVolumeClaims due to errors like: "Failed finding candidate datastores to place volume"
• TKC's or ClassyClusters fail to create when ephemeral volume mounts are configured on the Guest Cluster.
• When describing the PVC from Guest or Supervisor Cluster, users will see the following status:

# kubectl describe pvc -n test-namespace test-cluster-mzs2s-2bbs9-containerd

Name: test-cluster-mzs2s-2bbs9-containerd
Namespace: test-namespace
StorageClass: tanzu-storage-policy
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.example.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Mounted By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 3m50s (x43 over 13m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "csi.vsphere.example.com" or manually created by system administrator
Normal Provisioning 18s (x11 over 13m) csi.vsphere.example.com_42013b0be336cba14b73628e6f1f83e6_260e10a2-####-####-####-##########41 External provisioner is provisioning volume for claim "test-namespace/test-cluster-mzs2s-2bbs9-containerd"

Warning ProvisioningFailed 18s (x11 over 13m) csi.vsphere.example.com_42013b0be336cba14b73628e6f1f83e6_260e10a2-####-####-####-##########41 failed to provision volume with StorageClass "tanzu-storage-policy": rpc error: code = Internal desc = Failed finding candidate datastores to place volume. Error: No candidates datastores found in the Kubernetes cluster
 

• The vsphere-csi-controller logs report messaging like:

{"level":"error","time":"2023-08-11T06:56:51.376506933Z","caller":"wcp/controller.go:336","msg":"Failed finding candidate datastores to place volume. Error: No candidates datastores found in the Kubernetes cluster"
 

Environment

VMware vSphere 7.0 with Tanzu
VMware vSphere 8.0 with Tanzu

Cause

This problem occurs when a single ESXi host is added to the vSphere cluster on which Supervisor Cluster has been deployed and the new host is missing access to the Datastore referenced by the vSphere with Tanzu Namespaces.

See Cormac Hogan's blog or details on this. 

NOTE: This problem will occur even if the hosts that don't have access to the required datastore are placed into Maintenance Mode.

Resolution

This condition is expected if the Prerequisites for vSphere with Tanzu storage policies are not met. See documentation here

Workaround:
Review datastores tagged for use by the Storage Policy used by vSphere with Tanzu. Ensure these datastores are added to all ESXi hosts in the vSphere Cluster on which vSphere with Tanzu was enabled. If any hosts are missing the datastore(s) tagged for use in the Storage Policy, either add the datastore to those hosts, or remove those hosts from the vSphere Cluster.