Volume attach or detach operations fails with "time out for task before response from CNS" error.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Volume operations initiated by vSphere CSI driver may fail with a timeout error, even though the operation completes successfully in vCenter.
• Pods using Persistent Volumes backed by vSphere storage may fail to start or remain in a Container Creating state.
• The CSI controller reports errors: time out for task Task:<task-id> before response from CNS.
• To check vsphere-syncer logs, run:
  

  kubectl logs -n vmware-system-csi <pod-name> -c vsphere-syncer

  Err: time out for task Task:<task-id> before response from CNS"

• When describing an affected pod, the following event may appear:
  

  kubectl describe pod <pod-name>

  Events:
  Type     Reason              Age                   From                     Message
  ----     ------              ----                  ----                     -------
  Warning  FailedAttachVolume  59m (x2 over 61m)     attachdetach-controller  AttachVolume.Attach failed for volume "pvc-<pvc-id>" : rpc error: code = Internal desc = observed Error: "time out for task Task:<task-id> before response from CNS" is set on the volume "b5a#####-####-####-####-##########-<pvc-id>" on virtualmachine  "<node-name>"

Environment

• vSphere 8.0 Patch 06 (Update 3g) or earlier
• vSphere 9.0 with Supervisor not upgraded to version 9.0.2
• vSphere Kubernetes Service
• vSphere Supervisor
• vSphere CSI Driver - Vanilla Kubernetes Cluster

Cause

The issue is caused by a race condition in the CSI driver task monitoring implementation while waiting for Cloud Native Storage task completion

Resolution

To resolve this issue, perform the following upgrades based on your environment:

vSphere Supervisor

• For 8.x Environment: Upgrade vCenter Server to 8.0 Patch 07 (Update 3h) or later and upgrade the Supervisor cluster with builds shipped by 8.0 u3h patch releases. Both upgrades are required to fully resolve the issue. For Supervisor Version available on 8.0 u3h Product Interoperability Matrix
• For 9.x Environment: Upgrade the Supervisor cluster to version 9.0.2 or later. A vCenter upgrade is not required as the fix is delivered through an asynchronous Supervisor upgrade.  For Supervisor Version available 9.0 u2 Product Interoperability Matrix

Vanilla Kubernetes Clusters

• Upgrade the vSphere CSI Driver to release 3.6.0 or later.

Workaround

• If an immediate upgrade is not possible, restart the CSI controller. 
  
  Run the following command in the supervisor cluster to restart the CSI controller:
  
  kubectl rollout restart deployment csi-controller -n vmware-system-csi

Additional Information

This issue was also addressed in a future upstream vsphere-csi-driver version in Kubernetes - see: https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/3584 

• • Japanese version: TKC 仮想マシンまたは Service VM への PVC のアタッチが正常に動作せず、あわせて vSphere Namespace のイベントに 「ReconcileCnsVolumeMetadata: Failed to delete entry in CNS for instance with name」