Get error "SharedHost.getSharedDatastore: unable to find any shared datastores" when creating PVC with CSI

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

In TKGI with vSphere environment, When creating PVC with the storage class provisioned by vSphere CSI driver, the creation fails due to state "pending" and `kubectl describe pvc <NAME>` shows the error message as below:

Failed to provision volume with StorageClass "standard": SharedHost.getSharedDatastore: unable to find any shared datastores 

In CSI controller logs (on TKGI cluster master node), there would be error "No shared datastores found for nodeVm" as below:

E0524 04:57:41.434314   18322 controller.go:984] error syncing claim "######-#####-####-####-#####": failed to provision volume with StorageClass "demo-sts-storageclass": rpc error: code = Internal desc = failed to get shared datastores in kubernetes cluster. Error: No shared datastores found for nodeVm: VirtualMachine:vm-134545 [VirtualCenterHost: <VCENTER_FQDN>, UUID: ######-#####-####-####-#####, Datacenter: Datacenter [Datacenter: Datacenter:datacenter, VirtualCenterHost: <VCENTER_FQDN>]] 

Resolution

For Kubernetes cluster deployed by TKGI, the worker VMs could be across AZs, and each AZ can be connected with different datastores. However POD and PVC, it's not guaranteed in workers on specific AZ, thus when configure datastore for persistent volume, that datastore should be accessible from all worker VMs across multiple AZs. 

Datastore for Kubernetes persistent volume is configurable in two places: 

• In Kubernetes storage class > parameters > datastore
• If it's not specified in storage class, the datastore configured at "Ops Manager > TKGI > Setting > Kubernetes Cloud Provider > vSphere > Datastore Name" will take effect as default. 

It is mentioned in TKGI installation document, please guarantee that the datastore is not local to any single cluster in multi-AZ and multi-cluster environments. 

Note: For multi-AZ and multi-cluster environments, your Datastore Name should be a shared Persistent datastore available to each vSphere cluster. Do not enter a datastore that is local to a single cluster. For more information, see PersistentVolume Storage Options on vSphere.