Unable to delete VMFS datastore to reclaim storage space and getting error message : 'Error: The resource 'Datastore Name: <Datastore Name> VMFS uuid: xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx'

Products

• VMware vCenter Server

Issue/Introduction

Symptoms : 

• The data has been successfully migrated from the old VMFS datastore to the new VMFS datastore. As a result, the old VMFS datastore now needs to be deleted.
  
  
• Deleting VMFS datstore results in the following error :
  
  Error: The resource 'Datastore Name: Test-Datastore123 VMFS uuid: xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx' in use.
Cannot remove datastore 'Datastore Name: Test-Datastore123 VMFS uuid:
xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx' because "file system is busy". Correct the problem and retry the operation.

Steps to validate the issue: 

• Collect the VMFS datastore UUID and device backing information using the below command. 
  
   [root@Test221:~] vmkfstools -Ph -v1 /vmfs/volumes/Test-Datastore123
  VMFS-6.81 (Raw Major Version: 24) file system spanning 1 partitions.
  File system label (if any): Test-Datastore123
  Mode: public ATS-only
  Capacity 2 TB, 2 TB available, file block size 1 MB, max supported file size 64 TB
  Volume Creation Time: Mon May 21 12:49:19 2018
  Files (max/free): 16384/16372
  Ptr Blocks (max/free): 0/0
  Sub Blocks (max/free): 16384/16382
  Secondary Ptr Blocks (max/free): 256/255
  File Blocks (overcommit/used/overcommit %): 0/1474/0
  Ptr Blocks  (overcommit/used/overcommit %): 0/0/0
  Sub Blocks  (overcommit/used/overcommit %): 0/2/0
  Large File Blocks (total/used/file block clusters): 4102/0/134
  Volume Metadata size: 1537916928
  Disk Block Size: 512/512/0
  UUID: xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx
  Partitions spanned (on "lvm"):
          eui.xxxxxxxxxxxxxx:1
  Unable to connect to vaai-nasd socket [No such file or directory]
  Is Native Snapshot Capable: NO
  OBJLIB-LIB: ObjLib cleanup done.
  WORKER: asyncOps=0 maxActiveOps=0 maxPending=0 maxCompleted=0
  [root@Test221:~]
• Check if there is any extent mapping for the mentioned datastore. 
  
   [root@Test221:/var/log] esxcli storage vmfs extent list | grep -i Test-Datastore123
  Test-Datastore123                          xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx              0  eui.xxxxxxxxxxxxxx      
• Running voma check on the impacted VMFS datastore indicates there is a host still heatbeating.
  

  [root@Test221:/var/log] voma -m vmfs -f check -d /dev/disks/eui.xxxxxxxxxxxxxx:1
  Running VMFS Checker version 2.1 in check mode
  Initializing LVM metadata, Basic Checks will be done

  Checking for filesystem activity
  Performing filesystem liveness check..|Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
  Found 1 actively heartbeating hosts on device '/dev/disks/eui.xxxxxxxxxxxxxx:1'
  1): MAC address xx:xx:xx:xx:xx:xx, IP 10.x.x.121
     VOMA failed to check device : Device or Resource busy
  [root@Test221:/var/log]

Environment

VMware vSphere ESXi 8.x
VMware vSphere ESXi 7.x

Cause

Datastore deletion was not working since there is still 1 actively heartbeating host for the problematic datastore.

Resolution

If the host is reachable and accessible via vSphere Web client or Host UI, then refer to the below Broadcom Tech docs to unmount the data store via vSphere web client or host UI.

Refer :

• Unmount Datastores
• Unmount a Datastore in the VMware Host Client
• How to List, mount and unmount a volume/VMFS Datastore.

If the unmount option is not feasible, then we need to reboot the impacted host to release the lock on the problematic datastore.  Once the lock is released, then the datastore deletion task should work without any issues. 

Note: Before rebooting any Host, please make sure you have a good backup of the VM's running on that node, and if possible, please make sure you gracefully shut down the VM and then proceed with the host reboot. 

 

Additional Information

Also, refer to Error: "Device or Resource Busy" while attempting to delete a Datastore'.