Pods are unable to attach the volume with error "attachdetach-controller AttachVolume.Attach failed for volume: rpc error: code = Internal desc = Watch on virtualmachine timed out"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Pods in a Guest Cluster (e.g., Kafka pods) are stuck in ContainerCreating or Pending status. Describing the affected pods or the VolumeAttachment objects reveals the following error.
  
  Warning  FailedAttachVolume  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-uuid" : rpc error: code = Internal desc = Watch on virtualmachine "worker-node-name" timed out


• Logs of the CSI Attacher pod confirm the following error.
  
  failed to attach: rpc error: code = Internal desc = Watch on virtualmachine "worker-node-name" timed out


• CNS Manager Logs (vsanvcmgmtd.log) report that the volume is already attached to a different (often stale or powered-off) VM.
  
  error vsanvcmgmtd Failed to attach disk to vm: vm-<ID> with err: CNSVolumeAlreadyAttachedException (Volume is already attached to VM vm-<ID>)
  
  
• A worker node may be stuck in a Deleting state in the Supervisor Cluster but is already absent from the Guest Cluster's node list.

Environment

VMware vSphere Kubernetes Service

VMware vCenter Server

Cause

This issue occurs when a worker node is remediated or deleted by the Cluster API (CAPI) but fails to clean up its volume attachments correctly. This is often triggered by:

• Stale CNSNodeVMAttachment objects: The Supervisor Cluster still maintains a record that the volume is attached to the old, deleted, or powered-off VM.
• Snapshot Chain Depth: Excessive snapshots on the worker node's FCD (First Class Disk) can prevent vCenter from successfully modifying or destroying the VM, locking the volume to the stale hardware.

Resolution

The volume attachment must be manually decoupled from the stale VM/Node record before the new worker node can claim it.

Step 1: Manual Detachment in vCenter

1. Locate the stale worker VM in the vSphere Client inventory (the one identified in the vsanvcmgmtd.log).
   
   
2. Edit Settings and manually remove the affected hard disks (ensure "Delete files from datastore" is NOT checked).
   
   Note: If this fails with "XML element depth exceeds configured maximum," attempt the detach of the disks from the Host UI directly. Once the detach is complete, it  leads to the same getting successfully replicated on the vCenter Server UI.

Step 2: Clear Stale CNS Records on Supervisor

1. Log in to the Supervisor Cluster context.
   
   
2. Identify all cnsnodevmattachment entries associated with the stale node using the command below.
   
   kubectl get cnsnodevmattachments.cns.vmware.com -A | grep -i <stale-node-name>


3. Force-delete these stale records to release the CNS lock on the volume:
   
   kubectl delete cnsnodevmattachments.cns.vmware.com <attachment-name> -n <namespace> --force --grace-period=0

Step 3: Refresh CSI Driver

Restart the CSI driver components in the Guest Cluster to trigger a re-reconciliation of the volume attachments using the command below.

kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi

Step 4: Cleanup Stale VM

Once the volume has successfully attached to the new node and the pods are running, you can address the stale VM.