Changing an existing Input/Output Operations Per Second (IOPS) limit might cause a significant drop in I/O throughput of a virtual machine.

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

Symptoms:
Changing an Input/Output Operations Per Second (IOPS) limit might cause a significant drop in I/O throughput of virtual machines. When IOPS limit is changed based on the Storage I/O Control (SIOC) by using a Storage Policy Based Management (SPBM) policy, the VM performance might be significantly slower. Normally, when the SPBM policy is set, IOPS limits are handled by an I/O filter, while mClock, the default I/O scheduler, handles reservations and shares. Due to a logic fault, when existing IOPS limit is changed, I/Os might throttle at the mClock scheduler instead at the I/O filter. As a result, no I/Os get to the I/O filter which causes a significant drop in the I/O throughput of virtual machines.

Environment

VMware vSphere 7.0
VMware vSphere 8.0

Cause

When an existing SIOC policy is edited via SPBM, there could sometimes be an error which will result in incorrect application of policy. This could lead to a drop in VM IOPs performance.
 

Resolution

The issue has been resolved in ESXi 7.0.3 P09 (ESXi 7.0 Update 3q) and ESXi 8.0.3 (ESXi 8.0 Update 3).

1. Storage Policy with VC/IOPS capability, is configured in vCenter with SPBM Storage IO Control Settings like IOPS limit, IOPS reservation, IOPS shares.
2. VM disks using such storage policy show IOPS limit as per the IOPS settings on the Storage Policy in vCenter.
3. When the storage policy is being applied via SPBM the associated profiles for a particular VM can be queried using the following:
   • To query the profiles: PbmQueryProfile(queryProfile).
   • In case it is required to retrieve the contents of the profile, use PbmRetrieveContent(retrieveContent)
4. Starting ESXi 7.0.3 P09 (ESXi 7.0 Update 3q) and ESXi 8.0.3 (ESXi 8.0 Update 3), the limits are honored by the ESX default IO scheduler(mClock).
5. The default IO size for IOPs accounting is set to 32k in the ESX IO Scheduler (mClock). For IO sizes > 32K, a single IO can be considered multiple as the number of IOs is taken as (IO Size/32k), due to this the IOPs at the application will be lower. To handle this user can tune the IO size used for IOPs accounting by setting the advanced config option "/Disk/SchedCostUnit" which is set to 32K by default and can go upto a max of 256K.
   
   If IO size > 256K and user is only interested in using limits and not using reservation/shares, then user can set the advanced config option "/Disk/SchedulerWithReservation" to zero for just limits handling.