Modifying Queue Depth of NVMe-over-Fabrics controller/device
Article ID: 419899
Updated On:
Products
Issue/Introduction
Modifications to the queue depth should preferably be carried out in collaboration with the storage vendor.
ESXi 8.0 U3 and later builds shows the actual number of queues and queue sizes of all the controllers with the command 'esxcli nvme controller list':
Note: The method used to determine the number of IO queues and IO queue size might change in future. The command line parameters io-queue-size and io-queue-number only apply to vSphere 7.0u3 and later releases.
Environment
VMware vSphere 7.x
VMware vSphere 8.x
VMware vSphere 9.x
Resolution
There are two methods to adjust queue depth:
1. Change it globally by setting the module parameter vmknvme_io_queue_size. Host reboot is necessary for changes to take effect. The command is similar to:
esxcli system module parameters set -m vmknvme -p vmknvme_io_queue_size=<value>
2. Change it while running connect/discover command by setting the command parameter io-queue-size
esxcli nvme fabrics connect --io-queue-size=<value> <other parameters>
esxcli nvme fabrics discover --connect-all --io-queue-size=<value> <other parameters>
Example: esxcli nvme fabrics connect -a vmhba64 -i ###.###.###.### -p #### -s nqn.1986-03.com.ibm:nvme:2145.0000020######### --ioqueue-size=8 esxcli nvme fabrics discover -a vmhba64 -i ###.###.###.### -p #### -s nqn.1992-03.com.ibm:nvme:3342.0000030######### --ioqueue-size=8
Refer ESXCLI NVMe Commands for a full list of parameters.
The commands mentioned above can be used to set the maximum IO queue size and the maximum number of IO queues. The final results also depend on the capabilities of the NVMe-oF subsystem.
The --io-queue-size=<value> applies to all the IO controllers discovered and connected through discover command.
If both vmknvme_io_queue_size and io-queue-size are set, the later will overwrite the former.
These commands apply to all NVMe-over-Fabrics transport protocols.
Feedback
