Guest OS does not detect hot add, remove, or extend disk changes to the NVMe Controller

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

When you dynamically add, remove, or extend disk changes to the NVMe controller, users cannot see the changes in the guest OS.

Environment

VMware vCenter Server 6.7.x
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0.0
VMware vCenter Server 6.5.x
VMware vSphere ESXi 6.5

Resolution

Changing virtual disk configuration (hot add, remove and extend) during runtime (as dynamic namespace change) in NVMe spec v1.0 is not expected behavior in any guest OSes. So even our virtual device (vNVMe) supports hot add disk (and hot add controller) and extend, we do not recommend these operations. If customer wants to hot add vNVMe controller, add virtual disk together, and be aware that rebooting VM after any changes is recommended. Unload and reload device driver is a workaround as below:

 

• For Linux Guest OS, using RHEL7.1 as an example:

After dynamically adding a disk, unload and reload the driver using the command: rmmod nvme && modprobe nvme.
After dynamically removing a disk, unload and reload the driver using the command: rmmod nvme && modprobe nvme, or you can reset the controller driver using the command echo 1 > /sys/block/nvmeXn1pZ/device/reset where X is the Device Number and Z is the Partition Number.
After dynamically extending a disk, unload and reload the driver using the command: rmmod nvme && modprobe nvme, or you can reset the controller driver using the command echo 1 > /sys/block/nvmeXn1pZ/device/reset where X is the Device Number and Z is the Partition Number.

• For Windows Guest OS:

After dynamically adding a disk, unload and reload the driver.
After dynamically removing a disk, unload and reload the driver, or reset the driver with the command devcon.exe -r restart ‘nvme driver pci ID'. The system might automatically reboot to make the change effective.
After dynamically extending a disk, unload and reload the driver, or reset the driver with the command devcon.exe -r restart ‘nvme driver pci ID'. The system might automatically reboot to make the change effective.