NVMe -- Implementation within a vSphere Environment -- Quick Overview

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides a quick overview about topics around NVMe Implementation within a vSphere Environment ( NVMe Capable SAN )

This article does not include information related to the usage of NVMe devices in a vSAN Cluster (= the usage of NVMe with vSAN is dictated by following the vSAN Requirements ) 

You can use NVMe Storage within a vSphere Environment, 
while the NVMe HW incl. Driver and Firmware is provided by the HW Vendor. 

Environment

vSphere 8.x

Resolution

NVMe devices can be connected to the Host via various Transport Protocols, which dictate their implementation in order to work with vSphere. 
In order to get a better overview what that means, please review the Chapter "About VMware NVMe Storage" which includes the Requirements as well.

 

 

From a vSphere Perspective the most important factors for successful usage under vSphere are:

• NVMe HW needs to be compatible/certified with installed vSphere Version 
• NVMe Driver and Firmware need to be compatible/certified with installed vSphere Version and should reflect the latest compatible/certified versions.
• Having the latest vSphere Version installed on the ESXi Host(s) ensure that the latest fixes are applied on the vSphere side 

 

The Devices incl. Driver and Firmware can be checked for compatibility by using the VMware by Broadcom Compatibility Guide .
Alternatively, reach out to the HW Vendor to ask for certification status for implementing NVMe under vSphere.

Usually the HW Vendor can provide the latest Best Practices on how to install/upgrade the Driver and Firmware (since its their Product).
This will also ensure to get the latest information on HW functionality and Operation. 

 

 

 

Following a selection of available information on the Broadcom side, to provide a quick overview about implementing NVMe within a vSphere environment:

Architecture/Usage:

• NVMe with Fibre Channel - Official Broadcom Page
• NVMe-TCP - Guide for setting up in vSphere
• Using the "Memory Tiering over NVMe" feature in vSphere 8.0 Update 3
• esxcli nvme Commands
  
  

Selection of issues known to VMware by Broadcom ( also consult your HW Vendor about issues they are aware of): 

General: 

• NVMe over TCP ---- Devices encounter frequent path loss and performance degradation when data digest is enabled
• Issue with Unmap when using VMFS or VVOL datastores created out of NVMe devices
• WSFC with vVOL - Storage Tests may fail
• NVMe datastores disappear post ESXi FQDN change
• ESXi PSOD occurs with PsaNvmeAddControllerInt when an NVMe device fails to be added due to insufficient system's low memory

 

Depending on used HW: 

• PURE Storage --- ESXi hosts encounter numerous command aborts and timeouts on Pure Storage Arrays using NVME-oF
• PURE Storage --- Failed vMotion
• QLogic QLE2700/QLE2800 HBAs ---- Appear Offline After Upgrading to ESXi 8.x or Patching ESXi 7.x
• Cisco --- PSOD when using NVMe over Fiber Channel based VVOLs
• IBM --- Virtual Machine can not be migrated successfully from one NvmeTCP datastore  to another NvmeTCP datastore
• Infinidat ---- VMFS datastore creation on NVMe over TCP fails
• Hitachi ---- Recommendation for ESXi NVMe/TCP Configuration 
• Micron --- Repeated All-Paths-Down (APD) errors