VM with large I/O's cause disk group latency spike

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Elevated disk group latency observed on a single host within the vSAN cluster.
• On the ESXi host, the Monitor > vSAN > Performance view shows a sudden spike in the Delayed I/O Average Latency counter.

 

Environment

VMware vSAN 8.x

Cause

This is caused when one or more VMs are generating large I/O operations to a single VMDK or vSAN object. 

• In the vsan trace logs, the event DOMTraceOperationNeedsRetry is logged indicating more queueing at the down owner level—meaning more I/Os are waiting to be processed than the queue can handle:

2025-10-13T02:07:09.571340 [291729295] [cpu12] [4b69a9ba OWNER writeWithBlkAttr5 VMDISK] DOMTraceOperationNeedsRetry:4483: {'op': 0x45bc1c9d47c0, 'objUuid': '########-####-####-####-############', 'pendingUpdatesListLen': 128, 'inclusiveCommitCount': 0, 'prepWaitQueueLen': 0, 'status': 'VMK_LIMIT_EXCEEDED'}
2025-10-13T02:07:09.571343 [291729296] [cpu12] [4b69a9bc OWNER writeWithBlkAttr5 VMDISK] DOMTraceOperationNeedsRetry:4483: {'op': 0x45bc1cf8f580, 'objUuid': '########-####-####-####-############', 'pendingUpdatesListLen': 128, 'inclusiveCommitCount': 0, 'prepWaitQueueLen': 1, 'status': 'VMK_LIMIT_EXCEEDED'}
2025-10-13T02:07:09.571345 [291729297] [cpu12] [4b69a9be OWNER writeWithBlkAttr5 VMDISK] DOMTraceOperationNeedsRetry:4483: {'op': 0x45bc1cea9440, 'objUuid': '########-####-####-####-############', 'pendingUpdatesListLen': 128, 'inclusiveCommitCount': 0, 'prepWaitQueueLen': 2, 'status': 'VMK_LIMIT_EXCEEDED'}

Resolution

To achieve better throughput by splitting the workload across multiple smaller VMDKs. Each VMDK has its own queue, so distributing I/O across more objects increases the total available queue depth and improves overall performance. This reduces pressure on vSAN and helps lower latency.