Removing a LUN containing a datastore from VMware ESXi/ESX
search cancel

Removing a LUN containing a datastore from VMware ESXi/ESX

book

Article ID: 315416

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article provides steps for un-presenting a LUN containing a datastore from an ESXi host without removing the contents of the datastore.

Note: Deleting a datastore from vCenter Server and subsequent un-presentation of the LUN containing that datastore is effective most of the time. Removing a LUN without masking the LUN causes the host to continuously retry access to the LUN, and leads to an All-Paths-Down (APD) condition on the host. For more information, see VMware ESXi hosts in All-Paths-Down (APD) condition may appear as Not Responding in VMware vCenter Server.

Environment

VMware ESXi 6.5
VMware ESXi 6.7
VMware ESXi 7.0
VMware ESXi 8.0

Resolution

Un-presenting a LUN containing a datastore from host includes the following steps. Only execute Step 2 if you intend to discard the LUN in question. All other steps must be done.

  • Step 1: Identifying the volume in question
  • Step 2: Deleting the datastore when you do not want to preserve the data from the VMFS volumes for later use
  • Step 3: Masking the volume on each host when you want to preserve data from the VMFS volumes for later use or if the volume is already deleted
  • Step 4: Un-presenting the LUN from the array
  • Step 5: Rescanning all hosts
  • Step 6: Restoring normal claim rules on each host
  • Step 7: Final Rescan

Note: These steps must be performed using the ESX service console or the vSphere command line. These steps cannot be performed using the Manage Paths dialog in the vSphere Client.

Step 1: Identifying the volume in question

To identify the volume in question:

  1. From the vSphere Client, identify (by name) the datastore that you want to remove. If this is an RDM, go to Step 3.
  2. From an SSH session to the host, associate this name to an NAA identifier.

    To do this, run the command:

    esxcfg-scsidevs --vmfs

    Example output:

    mpx.vmhba0:C0:T0:L0:5 /dev/cciss/c0d0p5 4a9fd860-6098929b-70e1-001e0b1df518 0 Storage2
    naa.60a9800050334c424a6f55566d466f31:1 /dev/sdd1 4b5f0b60-a09c425d-3953-001e0b1df518 0 NetApp_iSCSI
    naa.6006016033201c00a43a4ab9be9cde11:1 /dev/sda1 4bc37846-18f3c301-2ecb-001e0b1df518 0 delete_datastore

    In this case, the datastore delete_datastore has naa.6006016033201c00a43a4ab9be9cde11, which is the NAA ID. You may also see an EUI ID, or a T10 ID, which are fully interchangeable.
     
  3. For RDMs:
     
    1. Edit the settings of the virtual machine in question, select the RDM you want to remove, and copy the vml.XXX number. This is the vml.ID to be used in the next step.
    2. Remove the RDM from the virtual machine and select Delete from disk before clicking OK. This destroys the mapping file, but not affect LUN content.
    3. On the host, run this command:

      esxcfg-scsidevs –u | grep –i vml.ID

      This command returns the NAA ID of that volume.
  4. From an SSH session to the host, associate this NAA ID with the vmhba identifiers.

    To do this, run the command:

    esxcfg-mpath –L | grep naa.ID

    For example:

    # esxcfg-mpath -L | grep -i naa.6006016033201c00a43a4ab9be9cde11

    Example output:

    vmhba1:C0:T1:L1 state:standby naa.6006016033201c00a43a4ab9be9cde11 vmhba1 0 1 1 NMP standby san fc.20000000c9739842:10000000c9739842 fc.50060160c1e0b7ec:5006016941e0b7ec
    vmhba1:C0:T0:L1 state:active naa.6006016033201c00a43a4ab9be9cde11 vmhba1 0 0 1 NMP active san fc.20000000c9739842:10000000c9739842 fc.50060160c1e0b7ec:5006016141e0b7ec
    vmhba0:C0:T1:L1 state:standby naa.6006016033201c00a43a4ab9be9cde11 vmhba1 0 1 1 NMP standby san fc.20000000c9739842:10000000c9739842 fc.50060160c1e0b7ec:5006016941e0b7ec
    vmhba0:C0:T0:L1 state:active naa.6006016033201c00a43a4ab9be9cde11 vmhba1 0 0 1 NMP active san fc.20000000c9739842:10000000c9739842 fc.50060160c1e0b7ec:5006016141e0b7ec

    In this example, the initial portion of the output, such as vmhba1:C0:T1:L1 and vmhba1:C0:T0:L1, breaks down to HBA1/0, Controller 0, Target (SP) 1/0, Lun 1.

    Note the portions that change (normally HBA and Target) between paths, and which stay the same (LUN).

    Some storage arrays use the same LUN number but change targets, such as EqualLogic and Lefthand. If you are not sure if your array uses differing targets or differing LUNs, DO NOT PROCEED and contact Broadcom Support.

Step 2: Deleting the datastore when you do not want to preserve the data from the VMFS volumes for later use

Note: For RDMs, skip this section. Similarly, skip this step if you intend to preserve the data on in the datastore in question. This step will destroy the partition table on the LUN.

To delete the VMFS datastore:

  1. Open the vSphere Client
  2. Open the Datastores view
  3. Right-click the VMFS datastore you want to remove click Delete.
If this fails, DO NOT PROCEED and contact Broadcom Support. For details, see Creating and managing Broadcom support cases .

 

 

Step 3: Masking the volume on each host when you want to preserve data from the VMFS volumes for later use or if the volume is already deleted

Note: This step must be done in all cases. There can be pending I/O on even a deleted volume, and proceeding directly to step 4 could allow a later APD event.

For this step, consider for example that the HBAs are changing. In this case, you need two rules to mask the volume, one for HBA1, and another for HBA2. You may use any rule number from 101 to 200. This example uses 192 and 193.

Before proceeding, ensure that you:
  • Verify that the datastore is not in use.
  • Unregister all objects stored on the datastore from the inventory, such as virtual machines and templates.
  • Verify that there are no third-party scripts or utilities running on a service console which could access the LUN in question.
  1. Connect to each ESX host using SSH.
  2. Run these commands:

    esxcli corestorage claimrule add --rule 192 –t location –A vmhba1 –L 1 –P MASK_PATH
    esxcli corestorage claimrule add --rule 193 –t location –A vmhba0 –L 1 –P MASK_PATH

    If your environment uses a single HBA (or Software iSCSI), you only need a single command.

    The -t location and -P MASK_PATH are mandatory. The other options are optional and are based on your environment. This example specifies both HBAs (vmhba0 and vmhba1), no controller (masking all controllers), no target (masking all targets), and LUN1 (masking only LUN1). If your storage uses targets to differentiate LUNs, you may want to specify the appropriate target with -T X. For more information, see the example at the end of this article.
     
  3. Run this command on each host to load the claimrule:

    esxcli corestorage claimrule load
     
  4. Run this command on each host and verify that the claimrule has loaded:

    esxcli corestorage claimrule list

    Example output:

    # esxcli corestorage claimrule list
    Rule Class Type Plugin Matches
    0 runtime transport NMP transport=usb
    1 runtime transport NMP transport=sata
    2 runtime transport NMP transport=ide
    3 runtime transport NMP transport=block
    4 runtime transport NMP transport=unknown
    101 runtime vendor MASK_PATH vendor=DELL model=Universal Xport
    101 file vendor MASK_PATH vendor=DELL model=Universal Xport
    192 runtime location MASK_PATH adapter=vmhba1 channel=* target=* lun=1
    192 file location MASK_PATH adapter=vmhba1 channel=* target=* lun=1
    193 runtime location MASK_PATH adapter=vmhba1 channel=* target=* lun=1
    193 file location MASK_PATH adapter=vmhba1 channel=* target=* lun=1
    65535 runtime vendor NMP vendor=* model=*

    Verify that each new rule has entries for show file and runtime. This indicates that the rules are saved and loaded. If you do not see this, DO NOT PROCEED and contact Broadcom Support.
     
  5. Run this command on all hosts to unclaim the volume in question:

    esxcli corestorage claiming reclaim –d naa.ID
     
  6. Confirm that steps 1-5 have been completed on all hosts.
     
  7. Run this command:

    In ESXi:
    grep vmkernel /var/log/messages

    In ESX:
    cat /var/log/vmkernel
     
  8. In the output of the command, locate entries similar to:

    Apr 12 16:18:03 ds-tse-h12 vmkernel: 4:03:31:44.853 cpu7:4109)WARNING: NMP: nmp_UnclaimPath: Physical path "vmhba1:C0:T1:L1" is the last path to NMP device "Unregistered Device". The NMP device has been unregistered.

    Warning: If you see any of these entries, DO NOT PROCEED and contact Broadcom support immediately for assistance:
  • Unable to unclaim path vmhba33:C0:T0:L1 on device naa.60a9800050334c424a6f55566d466f31. Some paths may be left in an unclaimed state. You will need to claim them manually using the appropriate commands or wait for periodic path claiming to reclaim them automatically.
    Note:  Reconfiguring the HA to use different datastore may help to resolve this issue. See  for more information   Configure Datastore Heartbeating
  • Apr 12 16:30:51 ds-tse-h12 vmkernel: 4:03:44:33.242 cpu5:4109)ScsiDevice: 1930: Can't unregister device naa.60a9800050334c424a6f55566d466f31 because it is in use. OpenCount:1 RefCount:2
  • Apr 12 16:30:51 ds-tse-h12 vmkernel: 4:03:44:33.242 cpu5:4109)WARNING: NMP: nmp_DeviceUnregister: Could not unregister NMP device "naa.60a9800050334c424a6f55566d466f31" Busy.

Step 4: Un-presenting the LUN from the array

After the LUN is masked from all the hosts, un-present the LUN from the SAN side. For more information, contact your storage array vendor.

Note: Ensure that the correct LUN is unpresented by verifying the NAA ID on the host and the array.
 

Step 5: Rescanning all hosts

To rescan all hosts:
  1. Connect to vCenter Server using the vSphere Client.
  2. Right-click the cluster and click Rescan for Datastores.

    Alternatively, navigate to Configuration> Storage adapters > Rescan on each host.
     
  3. To verify that the rescan was successful, run this command from all hosts:

    In ESXi:
    cat /var/log/messages |grep -i apd

    In ESX:
    egrep –i apd /var/log/vmkernel

    If this command returns any output, contact Broadcom Support immediately.
 

Step 6: Restoring normal claimrules on each host

To restore normal claimrules, perform these steps for every host that had visibility to the LUN, or from all hosts on which you created rules earlier:
  1. To remove the rule created in Step 3, run this command:

    esxcli corestorage claimrule delete --rule 192

    In this example, run this command for rules 192 and 193. Your environment may differ based on the number that you provided in Step3.
     
  2. Run this command to reload the claimrules:

    esxcli corestorage claimrule load
     
  3. Run this command and verify that the created rule is deleted:

    esxcli corestorage claimrule list

    Example output:

    # esxcli corestorage claimrule list
    Rule Class Type Plugin Matches
    0 runtime transport NMP transport=usb
    1 runtime transport NMP transport=sata
    2 runtime transport NMP transport=ide
    3 runtime transport NMP transport=block
    4 runtime transport NMP transport=unknown
    101 runtime vendor MASK_PATH vendor=DELL model=Universal Xport
    101 file vendor MASK_PATH vendor=DELL model=Universal Xport
    65535 runtime vendor NMP vendor=* model=*

    Note: You do not see the rules that you created earlier.
 

Step 7: Final Rescan

Perform a rescan on all ESXi/ESX hosts that had visibility to the LUN. If all of the hosts are in a cluster, right-click the cluster and click Rescan for Datastores. Previously masked LUNs should no longer be visible.
 




Additional Information

Appendix: Storage by Target

Some arrays use storage by target instead of storage by LUN. On a traditional array, such as Clariion and HP EVA, each controller is a target and individual volumes are specified by LUN number. On a target-based array, such as Equallogic and LeftHand, each volume is a separate SCSI target, each with LUN0. On these arrays, the output of the esxcfg-mpath appears similar to:

vmhba33:C0:T2:L0 state:active naa.6000eb395d4b82660000000000000010 vmhba33 0 2 0 NMP …
vmhba33:C0:T1:L0 state:active naa.6000eb395d4b82660000000000000012 vmhba33 0 1 0 NMP …
vmhba33:C0:T0:L0 state:active naa.6000eb395d4b82660000000000000014 vmhba33 0 0 0 NMP …

On each line of the output, you see L0 for LUN 0 and T0/1/2 for each of the actual volumes.

In this case, the masking commands are different and you must mask the target instead of the LUN. For example, use a command similar to:

esxcli corestorage claimrule add --rule 192 -t location -A vmhba33 -T 1 -P MASK_PATH

This command masks target 1, LUN 0.


All Paths Down for a storage device

VMware ESXi hosts in All-Paths-Down (APD) condition may appear as Not Responding in VMware vCenter Server

How to unmount a LUN or detach a datastore device from ESXi hosts

Creating and managing Broadcom support cases 

Powered by Wolken Software