vSphere CNS Volume Attach Fails with vim.fault.ResourceInUse for Kubernetes Pods

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• VMware vSphere Cloud Native Storage (CNS) volume fails to attach to a Tanzu Kubernetes Grid (TKG) or standard Kubernetes worker node. The workload pod remains stuck in the Init state and cannot successfully transition to Running.
  
  
• The attachdetach-controller repeatedly logs FailedAttachVolume events indicating a ResourceInUse fault from the vCenter API:

Warning  FailedAttachVolume  2m24s (x22 over 77m)  attachdetach-controller  (combined from similar events): AttachVolume.Attach failed for volume "pvc-<ID>" : rpc error: code = Internal desc = failed to attach disk: "<FCD_ID>" with node: "<NODE_UUID>" err failed to attach cns volume: "<FCD_ID>" to node vm: "VirtualMachine:vm-27595 [VirtualCenterHost: <REDACTED_HOSTNAME>,
UUID: <NODE_UUID>, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCent

Environment

• VMware Tanzu Kubernetes Grid Management 1.30.3

Cause

The First Class Disk (FCD) representing the Persistent Volume has a stale attachment and remains actively locked to the old worker node Virtual Machine in vCenter. Because the FCD is still mounted as a Hard Disk on the previous node, the vSphere CSI driver cannot acquire the lock to reassign and attach the disk to the newly scheduled target node.

Resolution

1. Log into the vSphere Client.

2. Navigate to Cluster > Monitor > Cloud Native Storage > Container Volumes.

3. Search for the volumeHandle (FCD ID) or the PVC name. Look at the Attached To column to identify the specific worker node VM holding the stale lock.

4. Right-click the offending worker node VM identified in the previous step and select Edit Settings.

5. Locate the Hard Disk that matches the volume (verify this by checking the disk file path to ensure it contains the FCD ID).

6. Click the X next to the Hard Disk to remove it. CRITICAL WARNING: Ensure that the checkbox to "Delete files from datastore" is UNCHECKED. Checking this will destroy application data.

7. Click OK to save the changes.

Once the disk is manually detached in vCenter, the vSphere CSI driver will automatically retry the attachment loop. Within 1-2 minutes, the driver will successfully attach the disk to the new worker node, allowing the pod to transition to a Running state.

Additional Information

For more information on Kubernetes volume attachment limits and vSphere CSI, refer to the VMware vSphere CSI Driver Documentation.