kubectl describe pod, the following error is observed in the events:Warning FailedAttachVolume #s attachdetach-controller AttachVolume.Attach failed for volume "pvc-###" : rpc error: code = Internal desc = observed Error: "time out for task Task:task-#### before response from CNS" is set on the volume "<UUID>" on virtualmachine "<VM-Name>"
This issue typically occurs due to a delay in the Kubernetes attach-detach controller process. When a pod is evicted (e.g., by a taint-eviction controller), the volume remains attached to the old node. The controller waits for the maxWaitForUnmountDuration (default is 6 minutes) before attempting a force-detach operation.
If the environment is running an outdated vCenter Server, the detach operation may fail or time out, leaving the volume "locked" or "in use" (ResourceInUse error). This prevents the volume from being successfully attached to the new node, resulting in the pod being stuck in ContainerCreating.
To resolve this issue, upgrade your vCenter Server and vSphere Supervisor to vSphere 8.0 Update 3h (Build 8.0.3.00700) or a later version.
This release includes critical fixes for the vSphere CSI driver that address improper volume detachment and CNS (Cloud Native Storage) communication timeouts.
Recommended steps:
csi-controller and csi-nodeagent logs).