Storage latency causes VM timeouts due to ESXi Round Robin evaluation timer (latencyEvalTime)

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi hosts configured with the VMW_PSP_RR (Round Robin) latency sub-policy may continue to send I/O to a degraded storage controller for up to 3 minutes. This behavior can lead to Guest OS timeouts, application hangs, or VM crashes if the storage array experiences high latency without a physical link failure.

Symptoms:

• High storage latency observed in vCenter/ESXi performance charts.
• Virtual Machines become unresponsive or report SCSI timeout errors.
• vmkernel.log contains warnings: WARNING: ScsiDeviceIO: 1780: Device naa.#### performance has deteriorated.
• Storage controller reports high cache pressure or "CMP Dirty Pages" limits reached.

Environment

VMware vSphere ESXi 8.x

VMware vSphere ESXi 7.x

Cause

The default ESXi latencyEvalTime is 180,000 milliseconds (3 minutes). During this window, ESXi locks onto a selected path and does not re-evaluate comparative latency across other paths. If a controller degrades "silently" (gray failure), ESXi continues using the path until the timer expires, which exceeds the standard Guest OS disk timeout of 30-60 seconds.

Resolution

Apply one of the following configurations to ensure faster path failover:

Option 1: Implement IOPS=1 (Recommended) Bypass the latency timer by switching paths after every I/O request. This is the industry standard for most modern storage arrays.

bash

esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device <device_ID>

Option 2: Reduce Latency Evaluation Timer Decrease the polling interval from 3 minutes to 3 seconds to detect degradation faster.

bash

esxcli storage nmp psp roundrobin deviceconfig set --type=latency --latency-eval-time=3000 --device <device_ID>