vSAN Storage Policy IOPS Limits: Expected Performance Degradation and Latency Spike within Guest OS
search cancel

vSAN Storage Policy IOPS Limits: Expected Performance Degradation and Latency Spike within Guest OS

book

Article ID: 446583

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

 

  • A virtual machine (VM) running on a vSAN cluster experiences sudden performance degradation, sluggishness, or increased I/O queuing inside the Guest Operating System.

  • Performance monitoring tools (e.g., vSphere Client, vRealize Operations/vCloud Suite, or Guest OS-level utilities) show a significant increase in vSCSI latency and storage queue lengths.

  • The physical underlying storage hardware (NVMe/SSD) displays normal latency and healthy performance metrics, showing no signs of hardware-level bottlenecks.

 

Environment

vSAN 8.x, vSAN 9.x 

Cause

This is expected behavior when a custom vSAN Storage Policy with an IOPS limit rule is assigned to a Virtual Machine Disk (VMDK) and the workload demands more I/O resources than the configured threshold.

When a VM meets or exceeds its defined IOPS limit:

  1. vSAN I/O Scheduler Intervention: The vSAN I/O scheduler throttles the traffic by artificially delaying the excessive I/Os to maintain the rate at or below the policy-defined threshold.

  2. Artificial Latency & Queuing: This intentional delay forces the remaining I/O requests to wait in the storage queue, manifesting as increased latency at the vSCSI and VM levels. The more a workload attempts to push past its threshold, the higher the artificial latency will spike.

  3. I/O Size Normalization: vSAN normalizes I/O measurements in 32KB increments. For example, a single 64KB I/O request is treated as two (2) normalized IOPS. Therefore, workloads processing larger I/O block sizes will hit the artificial limit much sooner than expected, even if the raw guest OS IOPS metric appears below the threshold.

To confirm if an object has an active IOPS limit policy enforced, log into an ESXi host in the cluster via SSH and check the vSAN object configuration using the following command:

esxcli vsan debug object list -u 6f1232ce-3665-4815-abd5-####### 

VSAN object UUID 

   6f1232ce-3665-4815-abd5-#######   << VSAN object UUID from affected ####.vmdk

Version:
  20
Owner:
  ####-###-
Policy:
  stripeWidth: 1
      cacheReservation: 0
      proportionalCapacity: 0
      hostFailuresToTolerate: 1
      forceProvisioning: 0
      spbmProfileId: ####-###-
      spbmProfileGenerationNumber: 0
      storageType: Allflash
      replicaPreference: Capacity
      iopsLimit: 5000    <<< 5000 IOPS limit with normalized IO of 32k
      checksumDisabled: 0
      subFailuresToTolerate: 1
      CSN: 2444
      SCSN: 1021
      spbmProfileName: ####-###-iol5000
      locality: None

Used:
  5864379383808
Used 4K Blocks:
  5864420605952
Size:
  2199023255552
Type:
  vdisk
Path:
  /vmfs/volumes/vsan:####-###-/####-###-/####.vmdk (Exists)

 

Note: As highlighted above, look for the iopsLimit flag under the policy output. If this value is explicitly defined (e.g., 5000), the vSAN storage tier will cap performance based on 32KB normalized IOPS blocks.

Resolution

Because this behavior is an intentional design of the vSAN Quality of Service (QoS) throttling mechanism, no technical defect exists. To alleviate the queuing and performance degradation within the Guest OS, choose one of the following management options:

Option 1: Increase or Remove the IOPS Limit

If the virtual machine is a critical workload and requires more throughput:

  1. In the vSphere Client, navigate to Policies and Profiles > VM Storage Policies.

  2. Select and edit the custom storage policy applied to the affected VM.

  3. In the Storage characteristics section, increase the value of the IOPS limit rule, or uncheck it entirely to remove restrictions.

  4. Save the policy. vSAN will apply the updated rule dynamically without requiring a virtual machine reboot.

Option 2: Distribute the Workload

If the limit must remain intact to prevent a "noisy neighbor" scenario from impacting the rest of the cluster:

  • Reschedule intensive application operations (such as backups, batch jobs, or database indexing) to off-peak hours.

  • Scale out the application layer to distribute the I/O demand across multiple virtual machines.

Alternative Verification via UI

  1. Go to the affected VM in the vSphere Client and select Monitor > vSAN > Performance.

  2. Drop down to the VMDK metric view.

  3. Compare the vSCSI IOPS against the Normalized IOPS. If the Normalized IOPS flatlines directly at your policy’s limit (e.g., 5000) while vSCSI Latency sharply rises, vSAN is actively enforcing the throttling rule.

Additional Information

Performance-metrics-when-using-iops-limit

 

Powered by Wolken Software