Error: "time out for task before response from CNS" during volume attach/detach

VMware vSphere Kubernetes Service VMware Telco Cloud Automation VMware Telco Cloud Platform

Volume operations initiated by vSphere CSI driver may fail with a timeout error, even though the operation completes successfully in vCenter.
Pods using Persistent Volumes backed by vSphere storage may fail to start or remain in a Container Creating state.
The CSI controller reports errors: time out for task Task:<task-id> before response from CNS.
To check vsphere-syncer logs, run:
kubectl logs -n vmware-system-csi <pod-name> -c vsphere-syncer
Err: time out for task Task:<task-id> before response from CNS"
When describing an affected pod, the following event may appear:
kubectl describe pod <pod-name>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedAttachVolume 59m (x2 over 61m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-<pvc-id>" : rpc error: code = Internal desc = observed Error: "time out for task Task:<task-id> before response from CNS" is set on the volume "#####-####-####-####-##########-<pvc-id>" on virtualmachine "<node-name>"

The issue is caused by a race condition in the CSI driver task monitoring implementation while waiting for Cloud Native Storage task completion

To resolve this issue, perform the following upgrades based on your environment:

vSphere Supervisor

For 8.x Environment: Upgrade vCenter Server to 8.0 Patch 07 (Update 3h) or later and upgrade the Supervisor cluster with builds shipped by 8.0 u3h patch releases. Both upgrades are required to fully resolve the issue. For Supervisor Version available on 8.0 u3h Product Interoperability Matrix
For 9.x Environment: Upgrade the Supervisor cluster to version 9.0.2 or later. A vCenter upgrade is not required as the fix is delivered through an asynchronous Supervisor upgrade. For Supervisor Version available 9.0 u2 Product Interoperability Matrix

Telco Cloud Automation

Vanilla Kubernetes Clusters

Workaround

If an immediate upgrade is not possible, restart the vmware-vsan-health service and the CSI controller.

Run the following command in the supervisor cluster to restart the CSI controller:
```
kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi
```
Run the following command to restart vmware-vsan-health from a vCenter ssh session:
```
service-control --restart vmware-vsan-health
```

This issue was also addressed in a future upstream vsphere-csi-driver version in Kubernetes - see: https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/3584