Error: "The operation is not allowed in the current state" while unmounting VMFS datastore
search cancel

Error: "The operation is not allowed in the current state" while unmounting VMFS datastore

book

Article ID: 446459

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms :

  • When attempting to unmount a VMFS datastore on an ESXi host, the operation fails with a "file system is busy" or "Device or Resource Busy" message.
  • vSphere client reports following error:

    "The operation is not allowed in the current state"



  • The /var/log/vmkernel.log displays errors similar to: WARNING: VC: 2445: unmounting opened volume ('datastore_uuid' 'datastore_name') with ref 2 is not allowed. VC: 2716: Unmount volume f532 28 2 4 1 0 0 0 0 0 : Busy
  • Command-line checks (lsof or vmkvsitools lsof) performed on the host show no open user-space file handles. The datastore browser shows only standard hidden VMFS system metadata files (e.g., .sdd.sf, .iormstats.sf).
  • LUNs corresponding to these datastores may have already been decommissioned at storage array end.
  • This behavior occurs frequently on snapshot LUNs or volumes managed by Site Recovery Manager (SRM).

Environment

  • VMware vSphere ESXi 7.x
  • VMware vSphere ESXi 8.x

Cause

  • Issue may also be observed due to stale dead paths for datastore in question.
  • Command esxcfg-mpath reports dead path:

    # esxcfg-mpath -b | grep -i <NAA-ID-of-Device>

    naa.############################## De11EMC iSCSI Disk (naa.##############################)
    vmhba#:C#: T#: L# LUN:# state:dead iscsi Adapter: Unavailable Target: Unavailable

  • A stale VMkernel file descriptor (ref 2 lock) persists in the hypervisor memory. The ESXi kernel's VMFS driver maintains an active reference to the volume, blocking the unmount operation.

Resolution

  1. CRITICAL: If the ESXi host is participating in a vSAN cluster or is prepared for NSX, restarting core management services on the fly can disrupt active data paths, LACP configurations, or storage availability. You must place the ESXi host into Maintenance Mode before proceeding with daemon restarts.

  2. Track host processes keeping descriptors open on the volume path to verify no active user-space file handles exist:

    1. Connect via SSH to the ESXi host.

    2. Execute: lsof | grep /vmfs/volumes/<Datastore_UUID>

    3. If no open files are returned, proceed to the next step. Ensure the current SSH terminal session is not parked within the volume path by executing cd /.

  3. Verify the physical block device world list to confirm only the default filesystem driver holds an open count:

    1. Execute: esxcli storage core device world list -d naa.<Device_ID>

    2. Confirm the output only shows idle0 under World Name. This indicates the ESXi kernel's VMFS driver holds a standard operational handle and no direct-access virtual machine utilities or RDMs are locking the device.

  4. Verify System Scratch, Syslog, and Coredump configurations are not targeting the datastore:

    1. Scratch Partition: Execute esxcli system option get -o /ScratchConfig/ConfiguredScratchLocation. If it references the datastore UUID, set a new path and reboot the host.

    2. Syslog Directory: Execute esxcli system option get -o /Syslog/global/logDir. If mapped to the datastore, modify the configuration to an alternate volume.

    3. Coredumps: Execute esxcli system coredump file list. If an active dump file exists on the volume, execute esxcli system coredump file remove --force.

  5. Confirm that Storage I/O Control (SIOC) and vSphere HA Datastore Heartbeating are not actively locking the volume:

    1. vSphere HA: In the vSphere Client, navigate to Cluster > Configure > vSphere Availability > Edit > Heartbeat Datastores. Ensure the datastore is explicitly excluded from heartbeating.

    2. SIOC: Navigate to Datastore > Configure > Settings > Storage I/O Control and confirm it is disabled.

  6. If the issue persists, attempt to clear stale daemon locks by executing the following commands via SSH on the affected host:

    1. /etc/init.d/storagerm restart

    2. /etc/init.d/vsantraced restart

    3. /etc/init.d/vsanepd restart

    4. services.sh restart

  7. If the ref 2 kernel lock remains after flushing the management agents, the orphaned kernel thread must be cleared by a system reboot. (The host should already be in Maintenance Mode from Step 5).

  8. Reboot the ESXi host.

  9. Attempt the datastore unmount operation again via the vSphere Client.

  10. Repeat the rolling reboot process for the remaining hosts in the cluster as necessary until the datastore is successfully unmounted.

Additional Information

Also, refer following article for "Device or Resource Busy' errors:

Error: "Device or Resource Busy" or "Device in use" when attempting to delete/unmount/disconnect a Datastore

Powered by Wolken Software