Pods Stuck in ContainerCreating Due to vSphere CNS Task Deadlock

Products

• VMware vCenter Server
• VMware Cloud Native Runtimes

Issue/Introduction

• In Kubernetes environments utilizing the vSphere CSI driver, administrators may experience an issue where a pod remains persistently stuck in the ContainerCreating state. This deployment failure is caused by an inability to attach the required Persistent Volume Claim (PVC) to the scheduled worker node. The underlying issue stems from a deadlock within the vCenter Cloud Native Storage (CNS) subsystem, which prevents the release of previous volume attachments.
  
  
• Administrators will observe failures at both the Kubernetes orchestration layer and the vSphere backend:
  
  
  • Kubernetes Pod Events (kubectl describe pod): Warning FailedAttachVolume AttachVolume.Attach failed for volume "pvc-######" : volume attachment is being deleted

  • Stuck VolumeAttachments: Querying kubectl get volumeattachment reveals older attachments stuck in a false attached state with pending finalizers.
    
    
• The vCenter /var/log/vmware/vsan-health/vsanvcmgmtd logs demonstrate a massive, unchanging backlog of conflicting CNS tasks (often hundreds of tasks deep):
  
  info vsanvcmgmtd [...] Update volume task conflicting with resource ########. 497 tasks are already in queue
info vsanvcmgmtd [...] Detach volume task conflicting with resource ########. 519 tasks are already in queue

Environment

• VMware vSphere Container Storage Interface
• VMware vCenter Server Cloud Native Storage

Cause

• The vSphere CNS component, which handles storage metadata and volume operations for Kubernetes, can experience a task serialization deadlock.
  
  
• During periods of high storage churn or if a pod/node is abruptly deleted, the CNS queue becomes overwhelmed with conflicting AttachVolume, DetachVolume, and UpdateVolumeMetadata tasks.
  
  
• As vCenter Server stops processing these tasks, the vSphere CSI controller in Kubernetes cannot finalize the deletion of stale VolumeAttachment objects, leaving the storage volume permanently locked.

Resolution

To safely clear the CNS task deadlock without impacting other workloads, you must restart the vsan-health service on the vCenter Server, which hosts the CNS component.

1. Open an SSH session to the vCenter Server Appliance (vCSA) and log in as root.

2. Execute the following command to force a restart of the service: vmon-cli -r vsan-health

3. Observe the Kubernetes cluster; the vSphere CSI controller will immediately drop the stuck queues, delete the stale VolumeAttachment objects, and successfully bind the volume to the new pod.