"FcdService AttachDisk already attached" Error when attempting to attach a persistent volume to a Kubernetes node

Products

• VMware vSphere ESXi

Issue/Introduction

This KB outlines the correct procedure to detach a persistent volume from a Kubernetes node via the Cloud Native Storage CnsVolumeManager

• May observe errors such as the following in the /var/log/vmware/vsan-health/vsanvcmgmtd.log on the vCenter Server:

2023-04-10T14:47:25.622Z error vsanvcmgmtd[38991] [vSAN@6876 sub=BlockService opId=ab70e9dd] CNS: Failed to attach disk to vm: vim.VirtualMachine:vm-123 with
 err: N3cns30VolumeAlreadyAttachedExceptionE(Volume ########-####-####-####-########420e is already attached to VM vm-456)

• Checking the referenced Virtual Machine 'vm-456' in the vSphere Client shows that it has the backing volume vmdk mounted to it as suggested by the error above.

Identify the VM correlating with the managed object ID 'vm-456' from the vCenter Server Managed Object Browser (MOB) at the following URL: https://vcenter_fqdn/mob/?moid=vm-456
Identify the volume vmdk by searching for the 'Volume Name' or 'Volume ID' in the vSphere Client, by selecting the relevant vSphere Cluster > Monitor > Cloud Native Storage > Container Volumes > "Add filter" searchbox

• Checking the events on PODs using the above volumes may show failures such as "MountVolume.WaitForAttach failed for volume "pvc-0123e456-1234-48d7-90g2-1234d04e69ce" : volume attachment is being deleted"

Cause

While the below steps outline the correct procedure to detach the volume from the unexpected Kubernetes node location, further investigation may be required into how Kubernetes and CNS became out-of-sync for the current volume location.

Resolution

• The VSAN MOB is required and will need to be enabled via the Ruby vSphere Console (rvc):

rvc

• When prompted for Host to connect to enter the following:

[email protected]@localhost

• After the Welcome to RVC prompt, enter the following command to start the vSAN MOB

vsan.debug.mob --start localhost
(It can be disabled/stopped afterwards if required by running vsan.debug.mob --stop localhost)

• Login to the CnsVolumeManager in the vSAN MOB via web browser at the following URL replacing <vc_fqdn> with the vCenter Server FQDN:

https://<vc_fqdn>/vsan/mob/?moid=cns-volume-manager

• Click on CnsDetachVolume and in the pop-up window which appears we need to enter the Volume ID and VM managed object reference in the VALUE field E.g. for the example in the symptoms above to detach the volume ########-####-####-####-########420e from the current VM vm-456 it would appear as follows:

<!-- array start -->
<detachSpecs>
   <volumeId>
      <id>########-####-####-####-########420e</id>
   </volumeId>
   <vm type="VirtualMachine">vm-456</vm>
</detachSpecs>
<!-- array end -->

• Click on the InvokeMethod to detach the volume from the VM

Additional Information

Impact/Risks:
If the detach of the persistent volume from a Kubernetes node is performed directly in the vSphere Web Client rather than via the CnsVolumeManager then it may introduce other issues as CNS will not be aware of the change applied to the First Class Disk (FCD) object.