Persistent volume provisioning fails with "unrecognised entity type" error in vSphere CSI

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• Provisioning a stateful set or a Persistent Volume provisioning via vSphere CSI driver fails and the pod gets stuck in "Pending" state.
  
  
• The CSI plugin is deployed with zoning in a topology aware Kubernetes cluster.
  
  
• Per the csi-provisioner logs, it is unable to fetch the hosts as a part of the topology awareness.
  
  E#### hh:mm:ss       # controller.go:###] error syncing claim "<ID>": failed to provision volume with StorageClass "<storage class name>": rpc error: code = Internal desc = failed to get shared datastores for topology segments [map[topology.csi.vmware.com/<type>:<value>]] in vCenter "<vCenter_fqdn>".
Error: failed to fetch hosts belonging to topology segment map[topology.csi.vmware.com/<type>:<value>]. Error: failed to fetch hosts from entity Datastore:datastore-<ID>. Error: unrecognised entity type found Datastore:datastore-<ID>.

E#### hh:mm:ss       # controller.go:###] error syncing claim "<ID>":failed to provision volume with StorageClass "<storage class name>": rpc error: code = Internal desc = failed to get shared datastores for topology segments [map[topology.csi.vmware.com/<type>:<value>]] in vCenter "<vCenter_fqdn>".
Error: failed to fetch hosts belonging to topology segment map[topology.csi.vmware.com/<type>:<value>]. Error: failed to fetch hosts from entity ResourcePool:resgroup-<ID>. Error: unrecognised entity type found ResourcePool:resgroup-<ID>.

Environment

• vSphere CSI Driver/Plugin 3.x
• Kubernetes versions 1.29 - 1.34
• vSphere 7.0 and 8.0

Cause

• The CSI driver is configured for topology awareness. The topology feature in vSphere CSI driver requires the topology labels to be applied on compute entities like vCenter root object, Datacenter, Folder, ClusterComputeResource, HostSystem, etc., and should not be applied on unsupported objects like Datastore, StoragePod, Resource Pools and Virtual Machines.
• This is because the CSI driver needs to find the shared datastore resources for all compute entities under a single valid topology tag. Applying labels to unsupported objects prevents the driver from identifying shared datastore resources.

Resolution

1. Log in to the vSphere Client.
2. Identify and locate the following inventory objects:
   
   1. Resource Pools
   2. Datastores and Datastore Clusters
   3. Virtual Machines
3. Remove any tags associated with the topology.csi.vmware.com category from these objects.
   Note: Do not apply any tags to a datastore that is actively linked to a VM Storage Policy used by the Kubernetes CSI driver.
4. Verify that the Kubernetes StorageClass only references the CSI driver and does not contain vSphere storage policy details.
5. Clear the driver cache by restarting the CSI controller deployment. Execute the following command from the Kubernetes cluster:
   kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi

Additional Information

This KB is only applicable for deployments where the CSI driver is configured for topology awareness.

Deploying vSphere Container Storage Plug-in with Topology