CNS volumes fail to mount on Tanzu Kubernetes Cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Pods are observed to be in ContainerCreating or Init state. 
  
  
• Following errors are seen upon describing the impacted pods:
  Warning  FailedAttachVolume  16s (x2 over 2m19s)    attachdetach-controller  AttachVolume.Attach failed for volume "pvc-###" : rpc error: code = Internal desc = Watch on cnsfileaccessconfig instance "<node_name>" timed out. Last seen error on the instance="Failed to query CNS volume: file:#### with error: rpc error: code = Internal desc = queryVolumeAsync failed for queryFilter: {{} [{{} file:####}] [] []  [] []   <nil> }. Err=time out for task Task:task-#### before response from CNS"


• Similar log entries are observed for csi-controller: 
  csi_handler.go:243] "Error processing" VolumeAttachment="csi-####" err="failed to attach: rpc error: code = Internal desc = Watch on cnsfileaccessconfig instance \"<node_name>\" timed out. Last seen error on the instance=\"Failed to query CNS volume: file:#### with error: rpc error: code = Internal desc = queryVolumeAsync failed for queryFilter: {{} [{{} file:####}] [] []  [] []   <nil> }. Err=time out for task Task:task-#### before response from CNS\""

Environment

vSphere Kubernetes Service

Cause

CSI controller is unable to query the volumes. 

Resolution

This issue has been resolved  in vCenter 8.0P07 and vCenter 9.0.2

Workaround

Restart the csi-controller deployment using the following command:

kubectl rollout restart deployment csi-controller -n vmware-system-csi

Additional Information

Japanese KB: Tanzu Kubernetes ClusterへのCNSボリュームのマウントが失敗する