Changing the queue depth for QLogic, Emulex, and Brocade HBAs

Products

VMware vSphere ESXi

Issue/Introduction

If the performance of your host bus adapters (HBAs) is unsatisfactory, or your SAN storage processors or heads are over-utilized, you can adjust your ESXi hosts' maximum queue depth values. The maximum value refers to the queue depths reported for various paths to the LUN. When you lower this value, it throttles the ESXi host's throughput and alleviates SAN contention concerns if multiple hosts are over-utilizing the storage and are filling its command queue.

Alternatively, if existing LUN queue depths are frequently fully utilized (leading to additional VM latency due to queued commands), it may, in some circumstances, be possible to increase the maximum LUN queue depth value. Important: The value should only be increased with the recommendation of your storage vendor and HBA/server vendor to ensure that the increased maximum IOPs will not have unintended performance impacts.

When one virtual machine is active on a LUN, you only need to set the maximum queue depth. When multiple virtual machines are active on a LUN, the Disk.SchedNumReqOutstanding value is also relevant. The queue depth value, in this case, is equal to whichever value is the lowest of the two settings: adapter queue depth or Disk.SchedNumReqOutstanding.

Note: The per-host parameter Disk.SchedNumReqOutstanding is deprecated in vSphere 5.5 to accommodate modification on a per-LUN basis. For more information, see Setting the Maximum Outstanding Disk Requests for virtual machines (1268).

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Resolution

This table lists the default Queue Depth values for QLogic HBAs for various ESXi/ESX versions:

NOTE: these are for your reference, any specific questions should be directed to the hardware vendor.

ESXi/ESX version	Queue depth
3.5 with driver version 6.04	16
3.5 with driver version 6.07 & 7.x	32
4.0	32
4.1	32
5.0	64
5.1	64
5.5	64
6.x	64

From the earliest versions of ESX/ESXi, the default Queue Depth value for Emulex adapters has been 32 by default, and because 2 buffers are reserved, 30 are available for I/O data.

From lpfc 12.8.317.0 (for ESXi 6.5 and 6.7) and 12.8.329.0 (for ESXi 7.0), the default queue depth is 64. See Emulex Drivers for VMware ESXi. Release Notes.

The default Queue Depth value for Brocade adapters is 32.

Notes:

VMware recommends keeping the HBA Vendors uniform across all hosts participating in a cluster. If you are using a mixed environment (for example, with QLogic and Emulex), VMware recommends keeping the Queue Depth value the same across all the hosts in the cluster.
If you are applying a host profile which includes Queue Depth settings and a host reboot is required after application, note that queue depth values can only be modified at boot time. Reboot the host and confirm that the Queue Depth values are set correctly. The capability to modify Queue Depth variables is reliant on a compatible driver, and some async drivers may not allow the Queue Depth to be set correctly.
For Broadcom adapters using bnx2fc driver, it is possible to control queue depth via bnx2fc_queue_depth parameter. For information on driver versions that support this parameter, see FCoE Driver Tuning for VMware ESXi, Linux, and Windows.
For Emulex adapters, starting 11.2 driver versions there are two separate drivers:

lpfc- used for FC adapters (LightPulse)
brcmfcoe- used for FCoE adapters (OneConnect)

Parameters names for brcmfcoe are the same as for lpfc.

For more information, see Emulex Software Kit Migration Guide.

Cisco VIC adapters: These adapters use fnic driver and the parameter is called fnic_max_qdepth. For more information, see Guide Cisco UCS fNIC Tunables.

ESXi 6.x and later

To adjust the queue depth for an HBA:

Verify which HBA module is currently loaded by entering one of these commands on the service console:
- For QLogic:
  
  # esxcli system module list | grep qln
- For Emulex:
  
  # esxcli system module list | grep lpfc
- For Brocade:
  
  # esxcli system module list | grep bfa
Run one of these commands:

Note: The examples show the QLogic and Emulex modules. Use the appropriate module based on the outcome of the previous step.
- For QLogic:
```
# esxcli system module parameters set -p ql2xmaxqdepth=64 -m qlnativefc
```
- For Emulex:
```
# esxcli system module parameters set -p lpfc0_lun_queue_depth=64 -m lpfc
```
- For Brocade:
```
# esxcli system module parameters set -p bfa_lun_queue_depth=64 -m bfa
```
Notes:
- In these commands, both ql2xmaxqdepth and lpfc0 use the lowercase letter L, "l", and not the numeric digit 1.
- In this case, the HBAs have their LUN queue depths set to 64.
- If all Emulex cards on the host must be updated, apply the global parameter, lpfc_lun_queue_depth instead.
Reboot your host.
Run this command to confirm that your changes have been applied:

# esxcli system module parameters list -m driver

Where driver is your QLogic, Emulex, or Brocade adapter driver module, such as lpfc, qlnativefc, or bfa.

The output appears similar to:

Name Type Value Description -------------------------- ---- ----- -------------------------------------------------- ..... ql2xmaxqdepth int 64 Maximum queue depth to report for target devices. .....

Resources

To set the Disk.SchedNumReqOutstanding value, see Setting the Maximum Outstanding Disk Requests for virtual machines (1268).
For more information about the default values and configuration options for the Emulex VMware drivers, see Emulex Drivers for VMware ESXi User Guide
For more information on controlling the LUN queue depth throttling, see Controlling LUN queue depth throttling in VMware ESXi/ESX (1008113).
For information on how to check the current values of the storage adapter and storage device queue depth, see Checking the queue depth of the storage adapter and the storage device (1027901).