How to detach a LUN device from ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides steps to unmount a LUN from an ESXi 6.x/7.x host, which includes unmounting the file system and detaching the datastore/storage device. These steps must be performed for each ESXi host.

Note: The documented method using the vSphere Client is limited to detaching the device on a per-host basis. To detach a storage device from multiple hosts at the same time, see Automating detaching datastores using PowerCLI and the vSphere SDK for Perl in the Resolution section in this article.

 

Symptoms:

• A datastore cannot be unmounted using the vSphere Client.
• The following error is reported:
  
  Cannot unmount volume Datastore_details because Reason. Correct the problem and retry the operation.
  
  Where reason can be any of these:
   
  • One or more virtual machines are still registered on it
  • Swap to host cache is enabled on it
  • Storage I/O Control is either enabled or running in statistics collection mode on it
  • File system is busy

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Resolution

Pre-unmount checklist

• If the LUN is being used as a VMFS datastore, all objects (for example, virtual machines, templates, and Snapshots) stored on the VMFS datastore must be unregistered or moved to another datastore.
  •  vCLS VMs will need to be migrated to another datastore or Retreat Mode enabled to safely remove the vCLS VM. See Unmounting or detaching a VMFS, NFS and vVols datastore fails (broadcom.com)
  • Note that vCLS VMs are not visible under the Hosts and Clusters view in vCenter
    
    
• All CD/DVD images located on the VMFS datastore must also be unmounted/unregistered from the virtual machines.
• Unregister the Virtual Machine templates stored on the datastore from the host. 

• The datastore is not used for vSphere HA heartbeat.
  
  
• The datastore is not part of a datastore cluster.
  
  
• The datastore is not managed by Storage DRS.
• The datastore is not configured as a diagnostic coredump Partition/File and Scratch Partition. For more information, see the following:
  • Configuring a diagnostic coredump partition on an ESXi host (broadcom.com)
  • Configuring ESXi coredump to file instead of partition (broadcom.com)
  • Creating a persistent scratch location for ESXi 8.x/7.x/6.x (broadcom.com)
    
    
• Storage I/O Control is disabled for the datastore.
  
  
• No third-party scripts or utilities running on the ESXi host can access the LUN that has issue. If the LUN is being used as a datastore, unregister all objects (such as virtual machines and templates) stored on the datastore.
  
  
• If the LUN is being used as an RDM, remove the RDM from the virtual machine. Click Edit Settings, highlight the RDM hard disk, and click Remove. Select Delete from disk if it is not selected and click OK.
  
  Note: This destroys the mapping file but not the LUN content.
  
  
• Check that there are no processes locking the VMFS with this command: esxcli storage core device world list -d

For example:

[root@esxi:~] esxcli storage core device world list -d naa.6589cfc00000008016##############

Device                                World ID  Open Count  World Name

------------------------------------  --------  ----------  ------------

naa.6589cfc00000008016##############     65575           1  idle0

naa.6589cfc00000008016##############     67003           1  sdrsInjector

naa.6589cfc00000008016##############     67080           1  storageRM

• Check if the LUN/datastore is used as the persistent scratch location for the host. For more information on persistent scratch, see Creating a persistent scratch location for ESXi 8.x/7.x/6.x (broadcom.com).

Note: When using the vSphere Client with vSphere 6.x and later, only these checks are performed during the datastore unmount operation:

• Host should not have any virtual machines residing on this datastore
  
  
• Host should not use the datastore for HA heartbeats

Obtaining the NAA ID of the LUN to be removed

From the vSphere Client

This information is visible in the Properties window of the datastore.

From the ESXi host

Run this command:
# esxcli storage vmfs extent list

For example:

esxcli storage vmfs extent list
Volume Name VMFS UUID Extent Number Device Name Partition
----------- ----------------------------------- ------------- ------------------------------------ ---------
datastore1 4de4cb24-########-####-##########f6  0   naa.6001c230d8abfe000f#############3
Storage2   4c5fbff6-########-####-##########f4     0   naa.6001c230d8abfe000ff#############1
Storage4   4c5fc023-########-####-##########f4   0   naa.6001c230d8abfe000ff7############1
LUN01      4e414917-########-####-##########f4  0   naa.60a98000572d54724a34##########1

Make a note of the NAA ID of the datastore to use this information later in this procedure.

Note: Alternatively, run the esxcli storage filesystem list command, which lists all file systems recognized by the ESXi host.

Unmount and Detach a LUN using the vSphere Client

Note: These steps to not apply to the vSphere Web Client or HTML5 client.

To unmount a LUN from an ESXi host using the vSphere Client:

1. If the LUN is an RDM, continue on step 2. Otherwise, in the Configuration tab of the ESXi host, click Storage. Right-click the datastore being removed and click Unmount.
   
   A Confirm Datastore Unmount window appears. When the prerequisite criteria have been passed, click OK.
    
2. Select a ESXi host and click the Storage view under Configure > Storage Devices:
   
   
   
   Note: Screenshot below is from a HTML5 vSphere Client environment.
    
3. Right-click the NAA ID of the LUN (as noted above) and click Detach. A Confirm Device Unmount window is displayed. When the prerequisite criteria are passed, click OK. Under the Operational State of the Device, the LUN is listed as Unmounted.
   
   Note: The Detach function must be performed on a per-host basis and does not propagate to other hosts in vCenter Server. If a LUN is presented to an initiator group or storage group on the SAN, the Detach function must be performed on every host in that initiator group before unmapping the LUN from the group on the SAN. Failing to follow this step results in an all-paths-down (APD) state for those hosts in the storage group on which Detach was not performed for the LUN being unmapped.
    
4. Confirm if the LUN is successfully detached. The LUN can then be safely removed from the SAN. For more information, contact your storage array vendor.
   
   
5. Perform a rescan on all ESXi hosts which had visibility to the LUN. The device is automatically removed from the Storage Adapters.

Note: If trying to unmount a LUN from a software iscsi based storage array see the the vSphere Documentation page before rescanning. 

 

When the device is detached, it stays in an unmounted state even if the device is re-presented (that is, the detached state is persistent). To bring the device back online, the device must be attached.

To permanently decommission the device from an ESXi host, manually remove the NAA entries from the host configuration:

1. To list the permanently detached devices, run this command:
   
   esxcli storage core device detached list
   
   Output will be similar to:
   
   Device UID State
   ------------------------------------ -----
   naa.50060160c46036df500############# off
   naa.6006016094602800c8e############# off
    
2. To permanently remove the device configuration information from the system, run this command:
   
   esxcli storage core device detached remove -d NAA_ID
   
   For example:
   
   esxcli storage core device detached remove -d naa.50060160c46036df500############

Unmount and Detach a LUN using the command line

To unmount a LUN from an ESXi 6.x/7.x host using the command line:

1. If the LUN is an RDM, skip to step 4. Otherwise, to obtain a list of all datastores mounted to an ESXi host, run this command:
   
   esxcli storage filesystem list
   
   The output which will list all VMFS datastores, will be similar to:
   
   Mount Point Volume Name UUID Mounted Type Size Free
   ------------------------------------------------- ----------- ----------------------------------- ------- ------ ----------- -----------
   /vmfs/volumes/4de4cb24-########-####-########f6   datastore1 4de4cb24-########-####-########f6    true    VMFS-5  140660178944 94577360896
   /vmfs/volumes/4c5fbff6-########-####-##########f4  Storage2 4c5fbff6-########-####-##########f4      true   VMFS-3  146028888064 7968129024
   /vmfs/volumes/4c5fc023-########-####-#########f4  Storage4 4c5fc023-########-####-##########f4    true    VMFS-3  146028888064 121057050624
   /vmfs/volumes/4e414917-########-####-########f4   LUN01 4e414917-########-####-##########f4     true     VMFS-5  146028888064 4266131456
   
    
2. To find the unique identifier of the LUN housing the datastore to be removed, run this command:
   
   # esxcfg-scsidevs -m
   
   This command generates a list of VMFS datastore volumes and their related unique identifiers. Make a note of the unique identifier (NAA_ID) for the datastore to be unmounted as this will be used later on.
   
   For more information on the esxcfg-scsidevs command, see Identifying disks when working with VMware ESXi (broadcom.com) .
   
   
   
3. Unmount the datastore by running this command:
   
   esxcli storage filesystem unmount [-u UUID | -l label | -p path ]
   
   For example, use one of these commands to unmount the LUN01 datastore:
   
   esxcli storage filesystem unmount -l LUN01
   esxcli storage filesystem unmount -u 4e414917-########-####-##########f4
   esxcli storage filesystem unmount -p /vmfs/volumes/4e414917-a8d75514-6bae-##########f4
   
   Note: If the VMFS filesystem to be unmounted has active I/O or has not fulfilled the prerequisites to unmount the VMFS datastore, an error will be seen in the vmkernel logs similar to:
   
   WARNING: VC: 637: unmounting opened volume ('4e414917-########-####-##########f4' 'LUN01') is not allowed.
   VC: 802: Unmount VMFS volume f530 28 2 4e414917############ f4ecf19b 4 1 0 0 0 0 0 : Busy
    
4. To verify that the datastore is unmounted, run this command:
   
   esxcli storage filesystem list
   
   Output will be similar to: 
   
   Mount Point Volume Name UUID Mounted Type Size Free
   ------------------------------------------------- ----------- ----------------------------------- ------- ------ ----------- -----------
   /vmfs/volumes/4de4cb24-########-####-##########f6 datastore1 4de4cb24-########-####-##########f6 true VMFS-5 140660178944 94577360896
   /vmfs/volumes/4c5fbff6-########-####-##########f4 Storage2 4c5fbff6-########-####-##########f4 true VMFS-3 146028888064 7968129024
   /vmfs/volumes/4c5fc023-########-####-##########f4 Storage4 4c5fc023-########-####-##########f4 true VMFS-3 146028888064 121057050624
   LUN01 4e414917-########-####-##########f4 false VMFS-unknown version 0 0
   
   The Mounted field is set to false, the Type field is set to VMFS-unknown version, and that no Mount Point exists.
   
   Note: The unmounted state of the VMFS datastore persists across reboots. This is the default behavior. To unmount a datastore temporarily, do so by appending the --no-persist flag to the unmount command.
    
5. To detach the device/LUN, run this command:
   
   esxcli storage core device set --state=off -d NAA_ID
    
6. To verify that the device is offline, run this command:
   
   esxcli storage core device list -d NAA_ID
   
   The output, which shows that the Status of the disk is off, will be similar to:
   
   naa.60a98000572d54724a##############
   Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d54724a##########)
   Has Settable Display Name: true
   Size: 1048593
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.60a98000572d54724a##########
   Vendor: NETAPP
   Model: LUN
   Revision: 7330
   SCSI Level: 4
   Is Pseudo: false
   Status: off
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Thin Provisioning Status: yes
   Attached Filters:
   VAAI Status: unknown
   Other UIDs: vml.020000000060a98000572d54724############################
   
   Running the partedUtil getptbl command on the device shows that the device is not found.
   
   For example:
   
   partedUtil getptbl /vmfs/devices/disks/naa.60a98000572d54724a##########
   
   Error: Could not stat device /vmfs/devices/disks/naa.60a98000572d54724a3########## - No such file or directory.
   Unable to get device /vmfs/devices/disks/naa.60a98000572d54724a##########
    
7. If the device is to be permanently decommissioned, it is now possible to unpresent the LUN from the SAN. For more information, contact your storage team, storage administrator, or storage array vendor.
   
   
8. To rescan all devices on the ESXi host, run this command:
   
   esxcli storage core adapter rescan [ -A vmhba# | --all ]
   
   The devices are automatically removed from the Storage Adapters.
   
   Notes:
   • A rescan must be run on all hosts that had visibility to the removed LUN.
   • When the device is detached, it stays in an unmounted state even if the device is re-presented (that is, the detached state is persistent). To bring the device back online, the device must be attached. To do this via the command line, run this command:
     
     esxcli storage core device set --state=on -d NAA_ID
9. If the device is to be permanently decommissioned from an ESXi host, (that is, the LUN has been or will be destroyed), remove the NAA entries from the host configuration by running these commands:
    
   1. To list the permanently detached devices:
      
      esxcli storage core device detached list
      
      Output will be similar to:
      
      Device UID State
      ---------------------------- -----
      naa.50060160c46036df500############# off
      naa.6006016094602800c8e############# off
       
   2. To permanently remove the device configuration information from the system:
      
      esxcli storage core device detached remove -d NAA_ID
      
      For example:
      
      esxcli storage core device detached remove -d naa.50060160c46036df500##############
   
   The reference to the device configuration is permanently removed from the ESXi host's configuration.
   
   Note: If the device is detached but still presented (step 7 was skipped), the preceding command fails to permanently remove the device from the system, and the device is automatically re-attached. Step 7 must be completed for the device to be permanently removed.

Notes:

• After detaching the LUN, it can be as not presented from the storage. However, if the esxcli storage core device detached remove -d NAA_ID command is run to permanently decommission the LUN from the ESXI host before removing the LUN from the storage, the LUN gets reattached to the host and must be detached again.
• While following these steps, vCenter storage connectivity alarms will still be seen. It would be advantageous to disable the alarm when carrying out these tasks.

Additional Information

If the LUN to be removed is flagged as perennially reserved, to prevent it causing host boot delays. The perennially reserved flag can prevent a successful unmount of the LUN.

 

To remove the perennially reserved flag, run this command to remove the flag:

 

esxcli storage core device setconfig -d $device --perennially-reserved=false

 

Then run this command to retry removal of the device:

 

esxcli storage core device detached remove -d $device

 

Note: There is no negative impact in setting the flag to false, even when it is not needed.

For more information about the perennially reserved flag, see ESXi host takes a long time to start during rescan of RDM LUNs (broadcom.com)

Identifying disks when working with VMware ESXi (broadcom.com)
ESXi host takes a long time to start during rescan of RDM LUNs (broadcom.com) 
Removing an inaccesible NFS datastore with SIOC enabled fails (broadcom.com)