CNS volumes fail to mount on Tanzu Kubernetes Cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Pods are observed to be in ContainerCreating or Init state. 
  
  
• Following errors are seen upon describing the impacted pods:
  Warning  FailedAttachVolume  16s (x2 over 2m19s)    attachdetach-controller  AttachVolume.Attach failed for volume "pvc-###" : rpc error: code = Internal desc = Watch on cnsfileaccessconfig instance "<node_name>" timed out. Last seen error on the instance="Failed to query CNS volume: file:#### with error: rpc error: code = Internal desc = queryVolumeAsync failed for queryFilter: {{} [{{} file:####}] [] []  [] []   <nil> }. Err=time out for task Task:task-#### before response from CNS"


• Similar log entries are observed for csi-controller: 
  csi_handler.go:243] "Error processing" VolumeAttachment="csi-####" err="failed to attach: rpc error: code = Internal desc = Watch on cnsfileaccessconfig instance \"<node_name>\" timed out. Last seen error on the instance=\"Failed to query CNS volume: file:#### with error: rpc error: code = Internal desc = queryVolumeAsync failed for queryFilter: {{} [{{} file:####}] [] []  [] []   <nil> }. Err=time out for task Task:task-#### before response from CNS\""

Environment

vSphere Kubernetes Service

Cause

CSI controller is unable to query the volumes. 

Resolution

Restart the csi-controller deployment using the following command:

kubectl rollout restart deployment csi-controller -n vmware-system-csi