Supported scenarios of NVME hot plug/removal in ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi supports PCIE hot plug for NVMe devices.
There are two kinds of hot-plug scenarios supported by ESXi:

1. Orderly/Planned hot-plug: hot-plug operation is initiated via notification of hardware or software after all of required proprietary steps, such as unmounting datastore, unclaiming device, etc. are done.
2. Surprise hot-plug: Operation is initiated without prior notification. Such as hot-unplug device manually.

Note: 

• ESXi supports hot-plug for only NVMe device which is in normal mode, i.e. the device isn't configured as RDM, PCI passthrough, or used as OS installation volume.
• It is recommended to do orderly hot-plug whenever as possible. Supporting surprise hot-plug is precaution against incorrect user operation. 

Environment

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 8.0.x

Resolution

Best practices

• Choose a hot-plug capable slot and then insert NVMe device. Wait for power indicator being on, if the platform supports driver power control.
• Issue esxcli hardware PCI list and check if target nvme device is listed. Output is similar to following:

      0000:86:00.0
   Address: 0000:86:00.0
   Segment: 0x0000
   Bus: 0x86
   Slot: 0x00
   Function: 0x0
   VMkernel Name: vmhba3
   Vendor Name: xxxxxxxxx
   Device Name: xxxxxxxx PCIe SSD Controller
   Configured Owner: VMkernel
   Current Owner: VMkernel
   Vendor ID: 
   . . .

• Check readiness of SCSI devices for the NVMe device: run esxcli storage core device list. Output is similar to following:

  (t10.NVMe____Dell_Express_Flash_NVMe_PM1725_1.6TB_SFFB25C007192382500
Display Name: Local NVMe Disk (t10.NVMe____Dell_Express_Flash_NVMe_PM1725_1.6TB_SFFB25C007192382500)
Has Settable Display Name: true
Size: 1526185
Device Type: Direct-Access 
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/t10.NVMe____Dell_Express_Flash_NVMe_PM1725_1.6TB_SFFB25C007192382500
Vendor: NVMe    
Model: Dell Express Fla
...

• If NVMe device contains VMFS datastores, these datastores will be mounted automatically. Run 'esxcli storage filesystem list' to list all datastores claimed. 

Removal

• Make sure no IO is running on the NVME device to be removed before starting hot-plug, no matter orderly or surprise.
• On orderly hot-unplug scenario,  user needs to run proprietary steps before unplugging. They are:
  • Power off VMs, if any.
  • Unmount VMFS and detach devices. For more information see vSphere support for hot-swap of local disks.
• In surprise hot-unplug scenario, user stop IOs running on NVMe device and then plug device out immediately not running the above steps.
• Hot-unplug device. Check that the device is removed from the system device listing:
  • Issue "esxcli storage core device list", check that the device is not listed.
  • Issue "esxcli storage nmp path list", check that there is no path for this device.
• Plug in the device, check that the device becomes available. Steps are:
  1. If the device was hot-unplugged orderly, it needs to re-enable device being detected during rescan via command "esxcli storage core claiming autoclaim --enabled=True".
  2. Issue "esxcli hardware pci list|grep -w <vmhba>", check that new adapter is listed.
  3. Issue "esxcli storage core device list", check that the device is listed.
  4. Issue "esxcli storage nmp path list", check that the path to device is listed.
• Steps to remount VMFS volumes:
  1. Mount VMFS volume: esxcli storage filesystem mount -l <datastore>
  2. List all VMFS volumes: esxcli storage filesystem list

Note: Do not perform hot-plug frequently, as they might lead to potential problems and corner case issues.