Unmounting or detaching a VMFS, NFS or vVols Datastore fails due to vCLS VMs

Products

• VMware vSphere ESXi

Issue/Introduction

When attempting to unmount or detach a VMFS, NFS, or vVols datastore, the operation fails with one or more of the following:

• vSphere Client Error: The resource 'Datastorename' is in use. Cannot unmount volume 'Datastorename' because "One or more virtual machines are still registered on it". Correct the problem and retry the operation.


• Via Command-line on the ESXi Host: Error when attempting to remove/unmount:  

Error unmounting /vmfs/volumes/####: Filesystem is busy (ref 11)

 

Via the vSphere Datastore Browser: One or more vCLS  folders/files is seen

Example:  

 

 

Environment

• VMware vCenter Server 
• VMware ESXi 

Cause

This issue occurs due to the presence of running vCLS Agent VMs on the affected Datastore.

Because these VMs maintain open file locks to ensure cluster services (DRS/HA), the ESXi Host is prevented from unmounting the volume.

 

Resolution

• Prior to unmounting or detaching a Datastore, ensure that vCLS VMs are moved or removed to release active file locks
• If the physical storage was removed before unmounting, place the cluster in Retreat Mode and reboot the ESXi host(s) to clear the stuck references

 

Step 1: Verification

1. Confirm no standard VMs or templates are registered to the datastore.
2. Verify that the scratch partition is not configured on this datastore.
3. If the UI shows no VMs but the error persists, use the ESXi CLI to check for outstanding worlds against the device backing the datastore to confirm vCLS locks.

 

Step 2: Clear vCLS Locks

Option A: Migrate vCLS VMs (Preferred) If there is another datastore attached to the hosts within the cluster, migrate the vCLS VMs:

1. Navigate to: Host Cluster > Configure > vSphere Cluster Services > Datastore.
2. Add a new datastore to the VCLS Allowed list.
3. Storage vMotion the vCLS VMs to the newly configured datastore.

Option B: Enable Retreat Mode If the target datastore is the only storage or if migration is not possible, use Retreat Mode to temporarily delete the VMs.

• For vCenter 7.0 U3o / 8.0 U2 and Later:

  1. Select the Cluster > Configure > vSphere Cluster Services > General.
  2. Select EDIT VCLS MODE and choose Retreat Mode.
  3. Click OK.

• For Versions Prior to 7.0 U3o / 8.0 U2:

  1. Select the vCenter Server > Configure > Advanced Settings.
  2. Click Edit Settings and set config.vcls.clusters.domain-c####.enabled to False (Replace #### with the cluster's domain ID from the URL).
  3. Click Save.

Note: If the datastore is shared between multiple clusters, Retreat Mode must be enabled on all clusters connected to that datastore. DRS will be non-functional while Retreat Mode is active.

Step 3: Unmount and Restore Services

1. Once the vCLS VMs are removed, perform the unmount or detach operation via the vSphere Client or CLI.
2. Revert the vCLS Mode to System Managed (or set the Advanced Setting back to True) to redeploy the vCLS VMs to available datastores and restore DRS functionality.

Additional Information

• Broadcom Techdocs - vSphere Cluster Services
• Disable vCLS on a Cluster via Retreat Mode
• Error: "Device or Resource Busy" or "Device in use" when attempting to delete/unmount/disconnect a Datastore
• Unmounting a datastore fails with error "Cannot unmount volume '<datastore name>' because "One or more virtual machines are still registered on it". Correct the problem and retry the operation."