vSphere support for hot-swap of local disks

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
vSphere ESXi supports different types of controllers for local disks, such as AHCI (SATA), SAS and RAID controllers, NVMe/PCIe SSDs connected to host PCIe bridge directly or behind Intel® Volume Management Device (VMD), etc. This article describes the preferred steps to perform hot plug and hot unplug (hot swap) operations on different types of local disks.

 

Note: This article does not cover hot swap in vSAN configurations. For more info on local disk operations for vSAN see VMware Docs: Replace a Capacity Device or VMware Docs: Replace a Flash Caching Device on a Host.

For further description of hot plug, hot unplug and hot swap concepts, see the Related Information section.

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 5.5
VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.5

Resolution

This section describes the supportability of different use cases for local disk hot-swap and recommended steps to perform during hot-plug/hot-unplug to avoid certain known issues.

Hot-Plug
 

vSphere ESXi will attempt to automatically discover hot-added SAS/SATA disks or NVMe/PCIe SSDs. In some cases, a manual re-scan may be required for the ESXi host to discover SAS or RAID HBAs. If ESXi does not automatically discover the hot-plugged disks, re-scan all storage adapters to trigger discovery. For more information about re-scanning storage adapters, see the vSphere Documentation.

Hot-Unplug
 

To avoid potential stability issues, disks should not be hot-removed if they are still in use by vSphere ESXi. Use these steps to ensure safe removal.
 

Notes:

• When performing this procedure, check for errors at every step.
• All steps should report success before moving on to the next.
• If any steps result in failure, then vSphere is still using the disk a system reboot will be required to remove or replace the disk.
• All steps must be completed successfully to ensure successful and supported hot-unplug operations.
• OEM vendors may recommend additional procedures for hot unplug in their storage configurations. Refer to OEM guidelines for additional recommendations and support statements.
• Steps 1, 3, 5 and 6 are not needed from ESXi version 7.0 and later.

1. Prevent devices from being re-detected during re-scan
   
   esxcli storage core claiming autoclaim --enabled=false;
    
2. Unmount the VMFS volumes on the device. For more information about unmounting VMFS volumes, see the Unmount Datastores section in the vSphere Documentation.
3. Detach the device. For more information about detaching SCSI devices, see the Detach Storage Devices section in the vSphere Documentation.
   
   Notes:
   • The above procedure may fail if the device is still in-use by some element of the system. If this occurs, reboot the ESXi host to remove the device.
   • Some NVMe/PCIe SSDs may present multiple namespaces. If this is the case, every namespace may have its own SCSI LUN registered. In these scenarios, every SCSI LUN registered from the NVMe/PCIe SSD device must be detached before the device can be unplugged.
      
4. Physically remove the device.
   
   Notes:
   • For SAS/SATA disks, vSphere may not automatically remove all reference the device following hot-unplug. If this occurs, rescan the system for storage devices. For more information about rescanning storage adapters, see the vSphere Documentation.
   • For PCIe/NVMe devices with multiple namespaces, all applicable vmhba devices should be unregistered after device removal. If any associated vmhba devices remain present following hot-unplug, reboot the ESXi host.
      
5. If desired, you may remove the disk or SCSI LUN from detached list with this command:
   
   esxcli storage core device detached remove -d device_identifier;
   
   For more information, see the vSphere CLI reference.
    
6. Re-enable autoclaiming with this command:
   
   esxcli storage core claiming autoclaim --enabled=true;

Hot-Swap

To perform a hot-swap operation, follow the Hot-unplug procedure and then follow the Hot-plug procedure.

Additional Information

• Hot Plug refers to the insertion of SAS/SATA disks to the enclosure connected to a supporting SAS/SATA HBA or RAID controller, or NVMe/PCIe SSDs in U.2 form factor to a PCIe enclosure while the ESXi host is running.
   
• Hot Unplug refers to the removal of SAS/SATA disks from the enclosure connected to a supporting SAS/SATA HBA or RAID controller, or NVMe/PCIe SSDs in U.2 form factor from a PCIe enclosure while the ESXi host is running.
   
• Hot swap refers to the replacement of a disk with a new disk while the ESXi host is running. Hot swap operations combine both hot unplug and hot plug operations.