Unable to Unmount VMFS Datastore with "Reason: Busy"

Products

• VMware vCenter Server
• VMware vSphere ESXi 7.0
• VMware vSphere ESX 8.x

Issue/Introduction

When attempting to unmount a VMFS datastore using the `esxcli` command, the operation fails with the message:  
`Volume 'Datastore-Name' cannot be unmounted. Reason: Busy`

[root@server01:~] cd /vmfs/volumes/Datastore-Name/

[root@server01:/vmfs/volumes/########-########-####-############] 

[root@server01:/vmfs/volumes/########-########-####-############] esxcli storage filesystem unmount --volume-label=Datastore-Name
Volume 'Datastore-Name' cannot be unmounted. Reason: Busy

 

Environment

ESXi 7.0

vSphere 8.x

Cause

This error indicates that the datastore cannot be unmounted because there are currently active processes or operations utilizing it. 
Common reasons include:

1. Virtual Machines are accessing the datastore – A VM might be running or have a virtual disk located on the datastore.
2. Active file system locks – Processes like file management or backup tools could be accessing files on the datastore.
3. Storage-related operations – Operations like storage vMotion, snapshot creation, 
   or other storage maintenance tasks could be preventing the unmount operation.

Resolution

1. Identify active VMs on the datastore:
   Use the `esxcli` command to list VMs using the datastore:

   #esxcli vm process list
   
   This will show any running VMs that may have their virtual disks on the datastore. 
   If you find any VMs using the datastore, power them off or migrate them to a different ESXi or different datastore.

2. Check for active files or open files.
   You can use the following command to check if there are open files on the datastore UUID:

   #lsof | grep /vmfs/volumes/########-########-####-############
  
   This will list processes that have files open on the datastore. 
   If any World name like "sh" related processes are listed, you will need to identify and terminate them.
   As the Symptoms description, the datastore unmount failed due to root user locked "/vmfs/volumes/########-########-####-############" directory,
   and this issue can be resolved by change the shell working directory from VMFS root volume to other location:
   
   [root@server01:/vmfs/volumes/########-########-####-############] cd /

   [root@server01:~] esxcli storage filesystem unmount --volume-label=Datastore-Name

3. Check for ongoing storage operations:
   Check for any ongoing storage-related operations, such as storage vMotion or backup tasks, which could prevent unmounting. 
   These operations can be monitored in the vSphere Client.

• Check that there are no processes locking the VMFS with this command: esxcli storage core device world list -d

For example:

 

[root@esxi:~] esxcli storage core device world list -d naa.###############################

Device                                World ID  Open Count  World Name

------------------------------------  --------  ----------  ------------

naa.###############################     65575           1  idle0

naa.###############################    67003           1  sdrsInjector

naa.###############################     67080           1  storageRM

• Check if the LUN/datastore is used as the persistent scratch location for the host. For more information on persistent scratch, see Creating a persistent scratch location for ESXi 8.x/7.x/6.x (broadcom.com).
  Note: When using the vSphere Client with vSphere 6.x and later, only these checks are performed during the datastore unmount operation:
• if device world includes worker threads like (hostd, vpxa, storageRM) then just required agent service to release or kill thread agent. 

      /etc/init.d/hostd restart (replace agent accordingly) 

• A more comprehensive list of possible factors which might prevent datastore unmounting see the pre-check list in KB: KB:323128