Provisioning a persistent volume in cluster fails with "Error: failed to fetch hosts from entity Datastore:datastore-<ID>. Error: unrecognised entity type found Datastore:datastore-<ID>"

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• Provisioning a stateful set or a Persistent Volume provisioning via vSphere CSI driver fails and the pod gets stuck in "Pending" state.
  
  
• The CSI plugin is deployed with zoning in a topology aware Kubernetes cluster.
  
  
• Per the csi-provisioner logs, it is unable to fetch the hosts as a part of the topology awareness.
  
  E0227 11:31:02.650004       1 controller.go:957] error syncing claim "<ID>": failed to provision volume with StorageClass "<storage class name>": rpc error: code = Internal desc = failed to get shared datastores for topology segments [map[topology.csi.vmware.com/<type>:<value>]] in vCenter "xxx.xxx.xxx.xxx".
Error: failed to fetch hosts belonging to topology segment [map[topology.csi.vmware.com/<type>:<value>]]. Error: failed to fetch hosts from entity Datastore:datastore-<ID>. Error: unrecognised entity type found Datastore:datastore-<ID>.

Environment

vSphere CSI Driver/Plugin 3.x

Cause

The CSI driver is configured for topology awareness. The topology feature in vSphere CSI driver requires the topology labels to be applied on compute entities like vCenter root object, Datacenter, Folder, ClusterComputeResource, HostSystem etc and should not be applied on the entities like Datastore, StoragePod, and Virtual Machines. This is because the CSI driver needs to find the shared datastores resources for all compute entities under a single valid topology tag. Even associating a tag with the datastore shall lead to the CSI driver not being able to find it.  

Resolution

• Remove the topology labels associated with the Datastore and Virtual Machines.
  
  
• Do not associate a tag with the datastore that is normally used to link the datastore to the storage policy.
  
  
• The storage class should only reference the CSI driver and shouldn't have the details of the storage policy on the vSphere side.

 

Refer the following documentation on Deploying CSI plugin with Topology- https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/container-storage-plugin/3-0/getting-started-with-vmware-vsphere-container-storage-plug-in-3-0/vsphere-container-storage-plug-in-deployment/deploying-vsphere-container-storage-plug-in-with-topology.html

 

Additional Information

This kb is only applicable for deployments where the CSI driver is configured for topology awareness.