Pod stuck in container creating state due to volume attached to wrong worker VM node
search cancel

Pod stuck in container creating state due to volume attached to wrong worker VM node

book

Article ID: 435098

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service VMware vSphere ESXi

Issue/Introduction

  • Pods are stuck in the ContainerCreating state. Upon running kubectl describe pod, the following error is observed in the events:
Warning  FailedAttachVolume  #s attachdetach-controller  AttachVolume.Attach failed for volume "pvc-###" : rpc error: code = Internal desc = observed Error: "time out for task Task:task-#### before response from CNS" is set on the volume "<UUID>" on virtualmachine "<VM-Name>"

Cause

This issue typically occurs due to a delay in the Kubernetes attach-detach controller process. When a pod is evicted (e.g., by a taint-eviction controller), the volume remains attached to the old node. The controller waits for the maxWaitForUnmountDuration (default is 6 minutes) before attempting a force-detach operation.

If the environment is running an outdated vCenter Server, the detach operation may fail or time out, leaving the volume "locked" or "in use" (ResourceInUse error). This prevents the volume from being successfully attached to the new node, resulting in the pod being stuck in ContainerCreating.

Resolution

To resolve this issue, upgrade your vCenter Server and vSphere Supervisor to vSphere 8.0 Update 3h (Build 8.0.3.00700) or a later version.

This release includes critical fixes for the vSphere CSI driver that address improper volume detachment and CNS (Cloud Native Storage) communication timeouts.

Recommended steps:

  1. Upgrade: Ensure your vCenter and Supervisor components are updated to at least 8.0U3h (Build 8.0.3.00700).
  2. Verify: After the upgrade, monitor the pods for approximately 10–15 minutes to confirm they transition to a Running state.
  3. If the issue persists: If you are already on version 8.0U3h or higher and still experience this behavior, gather the following logs for further investigation by Broadcom Support:
  • Supervisor WCP support bundle.
  • Guest cluster support bundle (including csi-controller and csi-nodeagent logs).
  • vCenter support bundle covering the timestamp of the attachment failure.