Unmounting an inactive datastore fails with the error: Cannot remove datastore 'datastore_name' because Storage I/O Control is enabled on it

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

This article provides steps to work around the issue when unmounting an inactive datastore fails with the Cannot remove datastore 'datastore_name' because Storage I/O Control is enabled on it error.

Symptoms:

• Cannot unmount an inactive datastore.
• Unmounting an unavailable datastore fails.
• Cannot delete the inactive datastore.
• You see the error:
  
  The resource '' is in use.
  Cannot remove datastore 'datastore_name' because Storage I/O Control is enabled on it. Correct it and re-try the operation.
  
• The datastore LUN is not available and, therefore, cannot be re-added.
• The Storage Adapter indicates the Operational State of the device as Lost Communication.
• All Paths to the device are marked as Dead.
• In the /var/log/vmkernel.log file, you see entries simialar to:
  
  2011-09-27T16:57:44.869Z cpu0:2060)ScsiDeviceIO: 2316: Cmd(0x412400773a80) 0x2a, CmdSN 0xc3 to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x1 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
  2011-09-27T16:57:44.869Z cpu0:2919)BC: 1858: Failed to write (uncached) object '.iormstats.sf': Device is permanently unavailable
  
  

  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment. 　
  
  

Environment

VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware vCenter Server 5.0.x
VMware vCenter Server 5.1.x
VMware ESX 4.1.x
VMware ESXi 4.1.x Embedded
VMware ESXi 4.1.x Installable

Cause

This issue occurs if Storage I/O Control (SIOC) is enabled on a storage device.

If an unplanned Permanent Device Loss (PDL) is invoked in vSphere 5.x when Storage IO control enabled on a storage device (iSCSI device or FC), the ESXi host cannot remount the VMFS volume.

Resolution

To work around this issue:

1. Connect to each ESXi 5.x host to which the LUN is presented by using SSH.
2. Run this command to stop the SIOC service:
   
   /etc/init.d/storageRM stop
   
   
3. In the vSphere Client, select the host and then click the Configuration tab.
4. Click Rescan All.
5. After the rescan completes, run this command to restart the SIOC service:
   
   /etc/init.d/storageRM start

Note: If the issue persists, put the affected ESXi 5.x host into maintenance mode and then reboot the host.

Additional Information

Notes:

• If possible, try to re-present the datastore to the host. If this is successful, the VMFS volume remounts and you can disable SIOC via the datastore Properties dialog.
• If there are virtual machines located on the datastore when the unplanned PDL occurs, the virtual machines must be stopped and unregistered from vCenter Server.

Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.x and 6.x