High QAVG observed even when IOPS is below the configured limit

Products

• VMware vSphere ESXi

Issue/Introduction

• esxtop displays a high QAVG
  
  
  
  
  
• For example, even if a VM is configured with a 100 IOPS limit, the virtual disk statistics in esxtop may show the I/O being limited to only 10 IOPS.
  
  

Cause

The ESXi I/O scheduler (mclock) accounts for I/Os larger than 32KB as multiple I/O operations.

For example, if a virtual machine is configured with a 100 IOPS limit and the guest OS issues 320KB I/Os, it will appear to be limited at 10 IOPS, as shown in the example above.

 

Note:
In versions prior to ESXi 7.0 Update 3q and 8.0 Update 3, when IOPS limits were set via Storage Policy, the I/O was controlled by iofilter instead of the mclock scheduler. In this case, I/O size is not taken into account. (Please see KB 385258 for more details.)
Even in versions prior to ESXi 7.0 Update 3q and 8.0 Update 3, if the limit is configured as a per-virtual disk throughputCap (rather than via Storage Policy), the mclock scheduler is used, and I/O size will be considered.

 

Resolution

This is expected behavior.

User can tune this behavior using advanced configuration options as outlined in KB 313242.

Additional Information

Default ESX IO Scheduler of IOPS Limit was changed from iofilter to mclock in 7.0 update3q and 8.0 update3

Changing an existing Input/Output Operations Per Second (IOPS) limit might cause a significant drop in I/O throughput of a virtual machine