Adjusting Round Robin IOPS limit from default 1000 to 1

Products

VMware vSphere ESXi

Issue/Introduction

Some storage vendors require that we set Round Robin IOPs to 1 to improve latency. This document describes how to implement the configuration change.

Symptoms:

VMware ESXi hosts using iSCSI/FC/FC/FCoE storage experiences latency issues with no signs of latency on the SAN side. You may see errors such as: "WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa..............." state in doubt; requested fast path state update..."
When one half of the storage paths for a LUN go offline (during, patching of storage controller, patching storage switch firmware and so on) , VMs (RHEL, other clustering VMs) get rebooted with the following error: "Message from ####: The CPU has been disabled by the guest operating system. Power off or reset the virtual machine."

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

VMware ESXi Round Robin PSP (Path Selection Plug-in) uses a Round Robin algorithm to balance the load across all active storage paths. A path is selected and used until a specific quantity of data has been transferred. After that quantity is reached, the PSP selects the next path in the list.
The quantity at which a path change triggered is known as the limit.
ESXi Round Robin PSP supports two types of limits:
- IOPS limit: The Round Robin PSP defaults to an IOPS limit with a value of 1000. In this default case, a new path is used after 1000 I/O operations are issued.
- Bytes limit: The bytes limit is an alternative to the IOPS limit. The bytes limit allows for a specified amount of bytes to be transferred before the path is switched.
Adjusting the limit can provide a positive impact to performance and is recommended by some storage vendors to change IOPS limit to 1.
Example:
The default of 1000 input/output operations per second (IOPS) sends 1000 I/O down each path before switching. If the load is such that a portion of the 1000 IOPS can saturate the bandwidth of the path, the remaining I/O must wait even if the storage array could service the requests. The IOPS or bytes limit can be adjusted downward allowing the path to be switched at a more frequent rate. The adjustment allows the bandwidth of additional paths to be used while the other path is currently saturated.

Resolution

Adjusting the IOPS parameter

To adjust the IOPS parameter from the default 1000 to 1, run this command:

In ESXi 7.x and later...

Workaround 1:

for i in $(esxcfg-scsidevs -c | grep ####| awk '{print $1}') ; do esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device="$i" ; done

Where, #### matches the first few characters of your naa IDs.

To verify if the changes are applied, run this command:

esxcli storage nmp device list

You see output similar to:

Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=iops,iops=1,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba33:C1:T4:L0, vmhba33:C0:T4:L0

Note: You do not need to restart the host for the changes to take effect.

Workaround 2:

1. Create a custom claim rule which applies then also for new LUN’s (reboot required) but not by default to existing LUN’s. How are we getting the existing LUN’s now to the intended setting as well as future new LUN’s? First have a look how many system SATP claim rules are already in place with (the list is not complete):

~ # esxcli storage nmp satp rule list

Name Device Vendor Model Driver Transport Options

------------------- ------ ------- ---------------- ------ --------- --------------------------

VMW_SATP_ALUA LSI INF-01-00 reset_on_attempted_reserve

VMW_SATP_ALUA NETAPP reset_on_attempted_reserve

VMW_SATP_ALUA IBM 2810XIV

VMW_SATP_ALUA IBM 2107900 reset_on_attempted_reserve

VMW_SATP_ALUA IBM 2145

Rule Group Claim Options Default PSP PSP Options Description ---------- ------------- ----------- ----------- -------------------------------------------------------------------------- system tpgs_on VMW_PSP_MRU NetApp E-Series arrays with ALUA support system tpgs_on VMW_PSP_RR NetApp arrays with ALUA support system tpgs_on VMW_PSP_RR IBM 2810XIV arrays with ALUA support system VMW_PSP_RR system VMW_PSP_RR

They have all certain settings like using the default PSP of VMW_PSP_RR for all Netapp devices but they don’t have a tuned setting configured. To create a new SATP claim rule you need to know some information like:

- Vendor: your vendor
- Model: your vendor model
- Options: specific options how to react on LUN resets
- Claim Options: specific things for e.g. ALUA
- Default PSP: the wanted default PSP
- PSP Options: the IOPS setting

2. To then create the the PSP with a certain IOPS or byte setting. There are two ways how to get information about your devices. The following command gives you information about the specific devices. Let’s take our example again:

~ # esxcli storage core device list -d naa.####| grep 'Vendor\|Model'

- - Vendor: FreeBSD
  - Model: iSCSI Disk

The next esxcli command give the possibility to find specific settings about the device:

~ # esxcli storage nmp device list -d naa.####| grep 'Storage Array Type Device Config'

Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}}

3. After you gathered all the information you can create a custom claim rule:

~ # esxcli storage nmp satp rule add -s "Your_SATP" -V "Your_Vendor" -M "Your_Model" -P "VMW_PSP_RR" -O "iops=1"

4. Now reboot the ESXi host and all the devices with a custom SATP rule should have the intended setting without any custom config:

esxcli storage nmp device list -d naa.####

naa.####

Device Display Name: FreeBSD Local SCSI/iSCSI Disk (naa.####)

Storage Array Type: VMW_SATP_ALUA

Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}}

Path Selection Policy: VMW_PSP_RR

Path Selection Policy Device Config: {policy=iops,iops=1,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}

Path Selection Policy Device Custom Config:

Working Paths: vmhba32:C1:T0:L253, vmhba32:C0:T0:L253

Is USB: false

Additional Information

When a VMware ESXi hosts cluster contains Raw Device Mappings (RDMs) that are used for Microsoft Failover Clustering, SAN port congestion matters. Adjusting the Round Robin limit value to 200 IOPS may provide better failover results.
Changing IOPS from the default to a lower value with the Path Selection Policy Round Robin option is a performance tuning setting and can help load balance across the storage processors and HBA paths in a storage array. With IOPS=1000, 1000 I/Os are sent down a path before alternating to the next path. You could potentially run into a scenario were all LUNs could be sending 1000 I/Os down the same path at the same time leaving one Storage Processor in the array processing the I/O, while the other one(s) are in an idle state. By alternating the path after one I/O, you get a better balance of I/O across the paths and thus storage processors. You get even balance of I/Os across the paths. If saturation occurs, it is very short in duration and gives the Storage Process or more time to process the I/O.
You may automate this with PowerCLI to change IOPS limit per cluster for every LUN that has Round Robin multipathing policy. For more information, see the attached PowerCLI_-_change_RR_IOPS_per_clust0.txt file.

Note: Consult your storage vendor to recommend if this value should be changed in your environment.

Attachments

PowerCLI_-_change_RR_IOPS_per_clust0.txt get_app