Error: "Device or Resource Busy" or "Device in use" when attempting to delete/unmount/disconnect a Datastore

Products

• VMware vCenter Server
• VMware vSphere ESXi
• VMware vSphere ESX 8.x
• VMware vSphere ESX 7.x
• VMware vSphere ESXi 8.0

Issue/Introduction

One or more of the following Symptoms are observed:

• When Datastore(s) are browsed, it does not have any files or folders
• There are no files or directories present under /vmfs/volumes/DATASTORE_NAME
• Deleting a Datastore fails
• Deleting the partition table of this datastore does not resolve the issue

 

One or more of the following Errors observed when attempting to delete, unmount or disconnect a Datastore:

• "Device/Resource Busy" or "Device is in use"

• "The resource '###' is in use. Cannot unmount volume '########' because "One or more virtual machines are still registered on it. Correct the problem and retry the operation."

Example:

 

• The resource 'Datastore Name: ######## uuid: ########' is in use. Cannot unmount volume 'Datastore Name: ######## VMFS uuid: ########' because "file system is busy". Correct the problem and retry the operation.

Examples:

 

/var/log/vmkernel.log of one or more of the ESXi Host(s) is reporting that the Volume is open:

YYYY-MM-DDTHH:MM:SS.ZZ Wa(180) vmkwarning: cpu18:#### opID=1111)WARNING: VC: 2445: unmounting opened volume ('####') with ref 4 is not allowed.
YYYY-MM-DDTHH:MM:SS.ZZ In(182) vmkernel: cpu18:#### opID=1111)FSS: 9868: File clusterAgent.stderr, fhID: 70231, is opened by world: 45645 - clusterAgent
YYYY-MM-DDTHH:MM:SS.ZZ In(182) vmkernel: cpu18:#### opID=1111)VC: 2716: Unmount volume ####  0 0 0 0 0 : Busy

 

 

/var/log/vmkernel.log of one or more of the ESXi Host(s) is indicating that Datastore is used as logging location for their ESXi Logs:

Remark: See for remediation the section "The Datastore is being used as "Output Logging" (= Syslog.global.logDir )"

YYYY-MM-DDTHH:MM:SS.ZZ cpu23:#### opID=7777)WARNING: VC: 2366: unmounting opened volume ('####' 'DatastoreName') with ref 80 is not allowed.

YYYY-MM-DDTHH:MM:SS.ZZ cpu23:#### opID=7777)FSS: 10072: File syslog.log, fhID: ##, is opened by world: 123123123 - vmsyslogd

YYYY-MM-DDTHH:MM:SS.ZZ cpu23:#### opID=7777)FSS: 10072: File vmkwarning.log, fhID: ##, is opened by world: 123123123 - vmsyslogd

YYYY-MM-DDTHH:MM:SS.ZZ cpu23:####  opID=7777)FSS: 10072: File vobd.log, fhID: ##, is opened by world: 123123123 - vmsyslogd

YYYY-MM-DDTHH:MM:SS.ZZ cpu23:#### opID=7777)FSS: 10072: File vmkernel.log, fhID: ##, is opened by world: 123123123 - vmsyslogd

YYYY-MM-DDTHH:MM:SS.ZZ cpu23:#### opID=7777)FSS: 10072: File vpxa.log, fhID: ##, is opened by world: 123123123  - vmsyslogd

YYYY-MM-DDTHH:MM:SS.ZZ cpu23:#### opID=7777)FSS: 10072: File hostd.log, fhID: ##, is opened by world: 123123123 - vmsyslogd

YYYY-MM-DDTHH:MM:SS.ZZ cpu23:#### opID=7777)VC: 2636: Unmount volume #### 0 0 0 0 0 : Busy

 

Environment

VMware vSphere ESXi 8.x

VMware vSphere ESXi 9.x

Cause

The issue occurs if the Datastore being deleted is being actively used/referenced by the VMware ESXi Host. This could be caused by some items like:

• The Datastore is referencing an ISO image (or in case of older VMs Floppy image)
• The Datastore still has active VM/VM Template references however the underlying LUN has been unpresented from the storage side
• A Virtual Machine has a snapshot chain that references the Datastore.
• A Virtual Machine has an active Replication ongoing. (There may hbr.#### files and VM files are getting updated under the Datastore Browser. However, no VMs or Templates running on the Datastore.)
• The Datastore is part of the Heartbeat selection in HA configuration
• The Datastore is being used as the "persistent scratch location" for logging
• The Datastore is being used as a location for vCLS VM(s)
• The Datastore is configured for SIOC
• There is an active command line Window open connecting to the Datastore (e.g via SSH/Putty session on the Host) resulting in an active process holding on to the Datastore on the Host
• Host process(s) holding lock on datastore

 

Resolution

The Datastore is referencing an ISO image (or in case of older VMs Floppy image)

This issue occurs if the Datastore is hosting an ISO that is mapped to a virtual machine on a VMware ESXi host on which this Datastore is mounted.

Even if ISO file is deleted from the Datastore, the ISO link to the virtual machine is not broken. If datastore is being tried to delete there are errors 'Device or Resource Busy' seen.

To determine if this is the cause:

1. Browse to the vCenter Server Datastore Inventory.
2. Select the Datastore from the left pane, then click the Virtual Machines tab in the right pane.
   
   If there are virtual machine to which the ISO from this Datastore is linked, the virtual machine name is listed.

To unmap the ISO:

1. Right-click the virtual machine and choose Edit Settings
2. On the Hardware tab, click the CD/DVD Drive.
3. In the right pane, de-select the Datastore ISO file

• • Client Device is the default option

     4.Click OK

For more information, see Virtual Machine displays two datastores in Summary tab despite all its files being located on one datastore (2105343).

 

The Datastore still has active VM/VM Template references

The VM templates can be moved to another Datastore by converting the template to a VM, then migrating to different Datastore and then converting it back to a VM template.

Or, if the VM template is not required anymore:

1. Browse to the Datastore Inventory on the vCenter Server.
2. Navigate to the VMs option.
3. Select the VM Templates in Folders option. 
   
   
    
4. Right click on the VM template and select the Remove from Inventory option.
   
   
    
5. Once the above steps are completed successfully, there should be no more active references to the Datastore . Datastore deletion should not cause any issues.(or the Datastore entry would automatically be taken off the inventory).

 

A Virtual Machine snapshot chain is referencing the Datastore

This issue occurs when a Virtual Machine resided on the Datastore that is attempting to be removed which had a Snapshot taken, and then was migrated off of the Datastore.

Determine if any Virtual Machines that have previously resided, or had disks reside on, this Datastore are currently running on snapshots.

1. Browse to the vCenter Server Datastore Inventory.
2. Select the Datastore from the left pane, then click the Virtual Machines tab in the right pane.
3. Right-click on the Virtual Machine in question and select the Consolidation task to commit the snapshot chain and remove the dependency on the Datastore which is being attempted to remove.

 

A Virtual Machine Replication is ongoing on the Datastore which needs to be unmounted

Check:

1. Check for Replication Files: Verify if the replication files (hbr.#####) are present in the Datastore
2. Access the Site Recovery plugin and open Site Recovery
3. Navigate to the Replication Tab
4. Check if the respective VM is listed under replication

 

Resolve by one of the two options:

1. Reconfigure VM Storage: If the VM is confirmed to be in Replication, reconfigure the VM to use a different Datastore
2. Alternatively, adjust Replication Settings by removing and then re-adding the Replication back for the respective VM with another Datastore.

This action should release the lock on the Datastore, allowing it to unmount.

Reconfiguring/Removing the replication for VM on another Datastore will initiate the full sync for the VM.

 

The Datastore is part of the Heartbeat selection in HA configuration

How to disable Datastore Heartbeating

1. Right click Cluster object
2. Click Edit Settings
3. Click Datastore Heartbeating
4. Click "Use datastores only from the specified list"
5. Do not select any Datastores

 

The Datastore is being used as "persistent scratch location"

1. vSphere Client -> Select the ESXi Host in the inventory
2. Click the Configure tab.
3. Select System.
4. Select Advanced System Settings.
5. The setting ScratchConfig.CurrentScratchLocation shows the current location of the scratch partition.
6. In the ScratchConfig.ConfiguredScratchLocation text box, enter a new directory path that is unique for this host
7. For example, /vmfs/volumes/DatastoreUUID/DatastoreFolder.
8. Reboot the ESXi Host for the changes to take effect.

 

The Datastore is being used as "Output Logging" (= Syslog.global.logDir )

The datastore cannot be unmounted because the one or more of the ESXi Hosts connected to it, have it configured as location for their ESXi logs

1. vSphere Client -> Select the host in the inventory.
2. Click the Configure tab.
3. Select System.
4. Select Advanced System Settings.
5. Click Edit.
6. In the Syslog.global.logDir text box, enter a new directory path for logging.
7. For example, [Datastore_name]/logdir.
8. Repeat the steps for other hosts if configured with Same datastore for logging.

9. Unmount the ESXi host, select NFS Datastore -> right click -> select "unmount Datastore" -> select check box to check ESXi hosts -> click Unmount

 

The Datastore is being used as a location for vCLS VMs

• The Datastore is being used as a location for vCLS vms.
  
  Follow instructions as per Disable vCLS on a Cluster via Retreat Mode and enable retreat mode.
  vCLS vms will be removed so Datastore will be available for deletion.
  Disable retreat mode and new vCLS vms will be created on a different datastore.

1.  

 

The Datastore is configured for Storage I/O Control (SIOC)

1. Select Datastore in the vSphere Client inventory and click Configure
2. Select General
3. Edit Datastore Capabilities.
4. Select "Disable Storage I/O Control and statics collection"

 

 

There is an active command line Window open connecting to the Datastore (e.g via SSH/Putty session on the Host): 

Example:

 

--> Close the Session to terminate the associated process on the Host

 

There is an active process on one of the ESXi Host(s) connected to that Datastore which is holding a lock on the Datastore

1. Open SSH to the ESXi host and login using the root user

2. Identify the Datastore UUID by running 

esxcli storage vmfs extent list

3. Search for a lock by running

lsof | grep DATASTOREUUID

4. If a process is identified, stop the process using /etc/init.d/<proccess> stop, or use the kill command to stop the process via the PID. 

 

Additional Information

Summarized: Before removing a Datastore, ensure that:

• Unregister all the Virtual Machine or Templates on that Datastore
• Unmap any ISOs or floppy disks that are stored on the Datastore
• Remove the Datastore from the highest point on the Inventory (In the vCenter Server Datastores view)
• Verify that the ESXi Host(s) do not display the Datastore. If they do display the Datastore, click Refresh in the ESXi Host's Storage section
• To identify open files on the Datastore that are supposed to be deleted, but appear as open, run the following commands:
  To get the Datastore UUID of the Datastore to be deleted

esxcli storage filesystem list
Mount Point                                           Volume Name                                 UUID                                    Mounted  Type            Size          Free
----------------------------------------------------  ------------------------------------------  --------------------------------------  -------  ------  ------------  ------------
/vmfs/volumes/69b284af-2fee185e-88af-############     OSDATA-69b284af-2fee185e-88af-############  69b284af-2fee185e-88af-############        true  VMFSOS    8321499136    5065670656
/vmfs/volumes/22cc9b08-41455a35-ff84-############     BOOTBANK1                                   22cc9b08-41455a35-ff84-############        true  vfat      4293591040    4023123968
/vmfs/volumes/c08d8530-ae4224b8-9e80-############     BOOTBANK2                                   c08d8530-ae4224b8-9e80-############        true  vfat      4293591040    4293394432
/vmfs/volumes/vsan:52cf8df2d45141d7-############      vsanDatastore                               vsan:52cf8df2d45141d7-############         true  vsan    643376873472  521881020570

vmkvsitools lsof|grep <datastore uuid>

 

 

For additional information on removing Datastores see:

• How to detach a LUN device from ESXi hosts (2004605).
• Remounting a disconnected NFS datastore from the ESXi command line
• Unmounting a datastore fails with error "Cannot unmount volume '<datastore name>' because "One or more virtual machines are still registered on it". Correct the problem and retry the operation."
• Unmounting or detaching a VMFS, NFS and vVols datastore fails