Persistent volume attach failure with "FcdService AttachDisk already attached" error in VKS

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Following errors are observed in the /var/log/vmware/vsan-health/vsanvcmgmtd.log on the vCenter Server:

YYYY-MM-DDTHH:MM:SSZ error vsanvcmgmtd[38991] [vSAN@6876 sub=BlockService opId=##] CNS: Failed to attach disk to vm: vim.VirtualMachine:vm-123 with err: N3cns30VolumeAlreadyAttachedExceptionE(Volume ########-####-####-####-########420e is already attached to VM vm-456)

• Verifying the referenced Virtual Machine 'vm-456' in the vSphere Client shows that it has the backing volume vmdk mounted to it as suggested by the error above.
  
  
• Identify the VM correlating with the managed object ID 'vm-456' from the vCenter Server Managed Object Browser (MOB) at the following URL: https://vcenter_fqdn/mob/?moid=vm-456
  
  
• Identify the volume vmdk by searching for the 'Volume Name' or 'Volume ID' in the vSphere Client, by selecting the relevant vSphere Cluster > Monitor > Cloud Native Storage > Container Volumes > "Add filter" searchbox
  
  
• Checking the events on PODs using the above volumes may show failures such as "MountVolume.WaitForAttach failed for volume "pvc-########-####-####-####-########" : volume attachment is being deleted"

Environment

vSphere Kubernetes Service

Cause

A stale volume attachment in the vCenter CNS database prevents the CSI driver from mounting the volume to a new node.

Resolution

Detach the volume from the unexpected Kubernetes node location:

1. Enable the vSAN MOB by running the following command on the vCenter Server SSH session:
   
   rvc
   
   
2. When prompted for Host to connect to enter the following:
   
   [email protected]@localhost
   
   
3. After the Welcome to RVC prompt, enter the following command to start the vSAN MOB
   
   vsan.debug.mob --start localhost
   (It can be disabled/stopped afterwards if required by running vsan.debug.mob --stop localhost)
   
   
4. Login to the CnsVolumeManager in the vSAN MOB via web browser at the following URL replacing <vc_fqdn> with the vCenter Server FQDN and using the username and password found on the vCenter under /etc/vmware/wcp/.storageUser:
   
   https://<vc_fqdn>/vsan/mob/?moid=cns-volume-manager
   
   
5. Click on CnsDetachVolume and in the pop-up window which appears, enter the Volume ID and VM managed object reference in the VALUE field.
   For example, in the symptoms above to detach the volume ########-####-####-####-########420e from the current VM vm-456 it would appear as follows:
   
   <!-- array start -->
   <detachSpecs>
      <volumeId>
         <id>########-####-####-####-########420e</id>
      </volumeId>
      <vm type="VirtualMachine">vm-456</vm>
   </detachSpecs>
   <!-- array end -->
   
   
6. Click on the InvokeMethod to detach the volume from the VM

Additional Information

If the detach of the persistent volume from a Kubernetes node is performed directly in the vSphere Web Client rather than via the CnsVolumeManager then it may introduce other issues as CNS will not be aware of the change applied to the First Class Disk (FCD) object.