Support for NVMe Hot-Add in Ubuntu 24.04

Products

• VMware vSphere ESXi

Issue/Introduction

As organizations increasingly deploy performance-critical workloads on virtual infrastructure, VMware’s virtual NVMe (vNVMe) controller is emerging as the preferred option over traditional virtual SCSI (pvSCSI) due to its lower latency and higher IOPS potential. One of the most requested operational features is the ability to hot-add or hot-remove NVMe disks, enabling maintenance flexibility without VM downtime.

While hot-add/remove capability is documented and supported in specific Windows configurations (with support for Asynchronous Event Notifications [AEN] introduced in the NVMe 1.3 specification), there has been limited information regarding equivalent support in Linux guest operating systems

This article shares verified observations looking to adopt this capability on Ubuntu 24.04 virtual machines.

Environment

ESXi 8.0.3
VM hardware version 19 or later

Resolution

Based on internal testing, it has been demonstrated that it is possible to:

1. Hot-add a new vNVMe controller to a running Ubuntu 24.04 VM.
2. Attach a new virtual disk to the controller.
3. Detect the new disk from within the guest OS using standard Linux tools.

Commands used for verification (within the Guest OS):

Note: These commands are standard Linux guest operating system utilities, not VMware-provided tools. They are provided for basic verification of device detection within the guest OS only. For comprehensive command usage, advanced troubleshooting, or further details on these utilities, please refer to the official documentation from the guest OS vendor

• To check for NVMe controller and device presence after hot-add:

lsblk

• To inspect NVMe device details and check controller version:

nvme list

• To verify kernel message logs (for device detection):

dmesg | grep -i nvme

 

The tests confirmed that Ubuntu 24.04 recognizes newly added NVMe devices dynamically.

Despite these observations, VMware does not currently provide official support or documentation affirming full compatibility of vNVMe hot-add/remove operations in Ubuntu 24.04 or other Linux guest operating systems.

This is due to:

• Lack of end-to-end validation covering real-world workloads, error conditions, and stress testing.
• The guest OS kernel and NVMe driver stack are being managed and updated by the Linux distribution vendor, outside VMware’s scope.
• Risk of unpredictable behavior in production due to a lack of tested corner cases or regression coverage.

Note:

Given the lack of explicit guest OS behavior documentation from VMware, the following guidance is recommended:

• Guest OS Vendor Involvement
  Engage with the Linux distribution provider (e.g., Canonical for Ubuntu) to confirm support for NVMe hot-plug operations in your chosen kernel and distribution version.
  
  
• Use in Production
  While basic functionality may work in lab scenarios, full production readiness should be verified through:
• • OS vendor support statements
  • Long-term testing in pre-production environments

• Controller/Kernel Considerations
  Ensure that the Linux kernel version supports NVMe 1.3 features, particularly asynchronous namespace notification.

• Hardware Version
  Use VM hardware version 19 or later (preferably 21) to ensure support for vNVMe controllers in VMware.

 

 

Additional Information

While the combination of VMware vNVMe and Ubuntu 24.04 appears to support basic hot-add NVMe operations at the OS level, this feature is not officially supported by VMware for production. Customers seeking this functionality should perform thorough internal testing and work with the guest OS vendor to validate their setup.
VMware continues to collaborate with OS vendors and drive improvements in vNVMe functionality. As validation efforts mature, official documentation and support statements may evolve.