High latency on vSAN with NVMe disks on Tri-mode controller

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• vSAN cluster report high latency and degraded VM performance.
• In the vSAN cluster, when navigating to Monitor > vSAN > Performance, high latency is observed at the cluster level.

 

• NVMe devices are connected through Tri-mode controller.
• In the /var/log/vmkwarning.log file of the ESXi host, performance has deteriorated messages are logged for vSAN disks across the cluster:

2025-08-15T14:28:30.337Z Wa(180) vmkwarning: cpu24:2098448)WARNING: ScsiDeviceIO: 1780: Device eui.######################## performance has deteriorated. I/O latency increased from average value of 3073 microseconds to 62329 microseconds.

 

 

Environment

VMware vSAN 7.x
VMware vSAN 8.x

 

Cause

NVMe devices attached to Tri-mode controller are not a vSAN supported configuration.

Resolution

Use Tri-mode controllers only for SAS and SATA drives in vSAN environments.

NVMe drives are expected to connect to PCI-E without passing through a RAID controller. In cases where additional PCI-E lanes are needed to support dense server configurations, PCI Switches are a supported alternative. Refer to this article for more information : vSAN support of NVMe devices behind tri-mode controllers