Unmount/Deletion of datastore is failing with the error: Cannot unmount volume '<datastore name>'

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• Performing an Unmount/Delete of datastore from vCenter Server Appliance (VCSA) UI may fail with any of the below error: 

Error A: Cannot delete file [<datastore name>] sysSwap-hls-<uuid>.swp

OR

Error B: The resource '<datastore name>' is in use.
Cannot unmount volume '<datastore name>' because "One or more virtual machines are still registered on it". Correct the problem and retry the operation.

 

• VCSA: /var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDTHH:MM:SS warning vpxd[05192] [Originator@6876 sub=Vmomi opID=m3h8kqos-2884-auto-286-h5:70000397-a9] VMOMI activation LRO failed; <<5221048b-27a1-f415-22e1-b7e5f89634b4, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 37958'>>, FileManager, vim.FileManager.delete>, N3Vim5Fault16CannotDeleteFile9ExceptionE(Fault cause: vim.fault.CannotDeleteFile
YYYY-MM-DDTHH:MM:SS info vpxd[05192] [Originator@6876 sub=vpxLro opID=m3h8kqos-2884-auto-286-h5:70000397-a9] [VpxLRO] -- FINISH task-9038
YYYY-MM-DDTHH:MM:SS error vpxd[05192] [Originator@6876 sub=Default opID=m3h8kqos-2884-auto-286-h5:70000397-a9] [VpxLRO] -- ERROR task-9038 -- 5221048b-27a1-f415-22e1-b7e5f89634b4(52e6f5b5-c4dd-f758-b481-160bd72c05b8) -- FileManager -- vim.FileManager.delete: :vim.fault.CannotDeleteFile
Result:
(vim.fault.CannotDeleteFile) {
   faultCause = (vmodl.MethodFault) null,
   faultMessage = <unset>,
   file = "[<datastore name>] sysSwap-hls-<uuid>.swp"
   msg = "Received SOAP response fault from [<<io_obj p:0x00007ff5e80a4c20, h:54, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/<host ID>/vpxa>]: Delete
"
}
Args:

Arg datacenter:
'vim.Datacenter:<datacenter uuid>:datacenter-3'
Arg datastorePath:
"[<datastore name>] sysSwap-hls-xxxxxx.swp"
Arg fileType:
"File"

• Esxi: /var/log/hostd.log

YYYY-MM-DDTHH:MM:SS info hostd[2100959] [Originator@6876 sub=Solo.Vmomi opID=m3p4i53l-895882-auto-j79y-h5:70172996-ba-31-3695 user=vpxuser:domain\user] Throw vim.fault.ResourceInUse
YYYY-MM-DDTHH:MM:SS info hostd[2100959] [Originator@6876 sub=Solo.Vmomi opID=m3p4i53l-895882-auto-j79y-h5:70172996-ba-31-3695 user=vpxuser:domain\user] Result:
 (vim.fault.ResourceInUse) {
    faultMessage = (vmodl.LocalizableMessage) [
       (vmodl.LocalizableMessage) {
          key = "com.vmware.vim.datastore.error.unmountVmfsVolumeFailed",
          arg = (vmodl.KeyAnyValue) [
             (vmodl.KeyAnyValue) {
                key = "name",
                value = "<datastore name>"
             },
             (vmodl.KeyAnyValue) {
                key = "reason",
                value = "vmsRegistered"
             }
          ],
       }
    ],
    name = "<datastore name>",
    msg = "",
 }

Environment

VMware vSphere Esxi 

VMware vCenter server

Cause

This is caused due to same datastore configured in the virtual machine's swap file location or vCLS VMs are placed on the same datastore.

Resolution

In order to resolve the issue, perform the below steps based on the error

• Error A:
  
  • Log in to vCenter Server (via SSH) .Enable SSH on VCSA.
  • Execute the below command to identify the host details

/opt/vmware/vpostgres/current/bin/psql -U postgres VCDB -c "select id,dns_name,ip_address from vpx_host where id=<numerical value highlighted above>"

Sample Output:

# /opt/vmware/vpostgres/current/bin/psql -U postgres VCDB -c "select id,dns_name,ip_address from vpx_host where id=17"

 id |      dns_name       |  ip_address
----+---------------------+---------------
 17 |  esxi.domain.name   | x.x.x.x

• • Log in to the ESXi host via SSH. Refer to Enable SSH on ESXi host .
  • Execute the below command

esxcli sched swap system get

Sample Output:

[root@esxi:~] esxcli sched swap system get
   Datastore Active: false
   Datastore Enabled: false
   Datastore Name:
   Datastore Order: 0
   Hostcache Active: false
   Hostcache Enabled: true
   Hostcache Order: 1
   Hostlocalswap Active: true
   Hostlocalswap Enabled: true
   Hostlocalswap Order: 2

• • Set the host in Maintenance Mode from VCSA UI
  • To Disable system swap on host login to ESXi host client, Navigate to Manage >System >Swap > Select Disable.

   

• Error B:
  
  • Log in to vCenter Server (VCSA) UI
  • Click on the 'Datastores' Tab on the Inventory
  • Navigate to the Virtual Machines tab on the right pane
  • Validate if any Virtual Machines are listed   

                       - If listed, perform a Storage vMotion of the VM. There may be an ISO mounted from the datastore and it needs to be removed from VM.

                       - If not listed, proceed below

                                    - Log in to one of the ESXi hosts SSH which does have access to this datastore, Refer to Enable SSH on ESXi host .
                                    - Navigate to the affected datastore using the below command :

                                                  cd /vmfs/volumes/<datastore name>/

                                    - List the folders in the directory to confirm the vCLS folders are only present.
                                    - Enable Retreat mode on the cluster where the host accessing the same datastore is present. Refer to Enable retreat mode in the cluster.
                                    - Proceed with unmount of the datastore
                                    - To disable Retreat mode , Navigate to Cluster > Configure > vSphere Cluster Services > General >Edit vCLS mode > Enable 'System Managed'.