High read latency on Hybrid vSAN mechanical disks resolving with Stripe Width policy

Products

• VMware vSAN

Issue/Introduction

Virtual machines (VMs) running on a Hybrid vSAN configuration may experience severe read I/O latency exceeding 200 ms. This behavior typically occurs when high concurrent read demand targets virtual disk components residing on a single mechanical capacity drive (HDD). This article provides guidance on using vSAN Storage Policies to distribute I/O across multiple physical spindles to remediate queue saturation.

• Virtual machines report significant performance degradation for disk-intensive applications.
• Guest OS disk latency metrics show sustained spikes (e.g., 150 ms to 225 ms).
• vSAN Performance monitoring shows device latency (DAVG) on capacity HDDs spiking to 30–40 ms or higher.
• The Disk Group indicates saturated Outstanding I/O (OIO) on specific mechanical devices while other drives in the cluster remain underutilized.

Environment

• VMware vSAN 8.x
• Hybrid vSAN Deployment (SSD Cache Tier + HDD Capacity Tier)

Cause

In a Hybrid vSAN deployment, read operations that miss the SSD cache tier fall through to the mechanical capacity disks. Mechanical drives have physical spindle performance constraints (typically ~150–200 IOPS). When high throughput or IOPS demand is concentrated on components residing on a single HDD, the device queue saturates, causing cascading latency at the Distributed Object Manager (DOM) layer.

Resolution

To distribute the workload across multiple physical spindles and eliminate single-HDD bottlenecks, increase the stripe width of the virtual disk objects.

1. Identify the impacted VM and the specific host where the DOM Owner resides.
2. Review vSAN performance charts to confirm DAVG spikes on specific capacity HDDs (naa.####).
3. Navigate to Policies and Profiles > VM Storage Policies.
4. Create a new policy or edit an existing one used by the impacted VM.
5. In the vSAN structure, locate the Number of disk stripes per object (NumberOfDiskStripesPerObject) setting.
6. Set the value to 2 or 3. This splits each replica of the VMDK across multiple physical capacity drives.
7. (Optional) Configure a Flash Read Cache Reservation percentage to ensure the active read working set is prioritized in the SSD cache tier.
8. Apply the policy to the virtual machine.
9. Monitor the physical HDD DAVG to confirm the load is distributed and latency is reduced to healthy targets (10–20 ms for magnetic media).