Unmounting or detaching a VMFS, NFS and vVols datastore fails

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides the required steps to successfully complete datastore detach or unmount workflow, which might be impacted due to the placement of vCLS VMs.

Symptom :

• Unmounting a VMFS, NFS or vVols datastore fails.
  
  
• Unable to delete the datastore as vCLS VMs present in datastore.
  
  
• Error noticed while removing the datastore:
    " The resource 'Datastorename' is in use. Cannot unmount volume 'Datastorename' because "One or more virtual machines are still registered on it". Correct the problem and retry the operation. "
  
  Select host > Monitor > Tasks.
  
  

Verification:

• No VMs or VM Templates running on this datastore.

• Scratch partition is not configured on this datastore.

• Noticed vCLS folder and files under this datastore.
  
  

Environment

VMware vSphere vCenter 7.x

VMware vSphere vCenter 8.x

Cause

• This issue occurs due to the presence of running vCLS VMs on the datastores. Since the vCS VMs still refer to the datastore which needs to be unmounted, the datastore throws an error suggesting that it is still in use.

Resolution

• Prior to unmounting or detaching a datastore, check if there are any vCLS VMs deployed in that datastore. If there are any, migrate those VMs to another datastore within the cluster, if there is another datastore attached to the hosts within the cluster. 
  
  If the datastore that is being considered for "unmount" or "detach" is the only datastore connected to the host and this datastore has vCLS VM, then the only option is to remove the vCLS VM by using Retreat Mode on the cluster.

Note: This datastore might be connected to different hosts in different clusters and vCLS VMs from all these different clusters might be stored in this datastore. In such a case, Retreat Mode has to be configured on all the clusters connected to this datastore, so this datastore can be unmounted or detached. 

• • Use Retreat Mode to remove vSphere Cluster Service VMs from a cluster.  First, enable Retreat Mode in the cluster so that all the vCLS VMs in the cluster gets deleted. If in case the same datastore is shared between multiple clusters and the vCLS VMs from different clusters are placed in this same datastore then in all the clusters where the corresponding cluster vCLS VMs are placed in this datastore should be enabled with Retreat Mode. Also when this Retreat Mode is enabled, the DRS in that particular clusters will be non-functional until Retreat Mode is disabled back. Once the datastore Maintenance Mode is successful, Retreat Mode should be disabled in all clusters connected to this datastore where its enabled in order to get back the vCLS VMs in all the cluster for DRS functionality. 
    
    
• Alternatively, you can also add the new datastore to the vSphere Cluster Services - allowed datastores list. After adding the allowed datastores, you can then Storage vMotion the vCLS VMs to the newly configured datastores which will in turn free up the datastores that you wish to decommission.

You can find this option under path: Host Cluster > Configure > vSphere Cluster Services > Datastore > VCLS allowed - Add :  new datastore

 

Additional Information

Reference Documentation: vSphere Cluster Services

Note: If the storage backing the datastore has been removed before the datastore was unmounted and removed and vCLS VMs are still running on it the only solution is to place the cluster(s) in Retreat Mode and reboot the host(s) with the stuck vCLS VMs. Once rebooted the vCLS VMs will be gone and datastore removal may proceed if still present (typically the datastore will also be removed already as no references remain).

Disable vCLS on a Cluster via Retreat Mode

vSphere Cluster Services (vCLS) in vSphere 7.0 Update 1 and newer versions