Applying QoS on a specific NVMe over TCP datastore throttles non-targeted NVMe over TCP datastore traffic as well.

Products

• VMware vSphere ESXi

Issue/Introduction

VMware introduced the support of NVMe over TCP storage adapters in 7.0U3.  Should NVMe over TCP performance considerations need to be managed and prioritized, upstream physical network and storage array mechanisms are leveraged.  A typical mechanism for this workload prioritization is applying QoS (Quality of Service) on the NVMe over TCP traffic at the storage array level. 

The methodology for applying QoS to this type of traffic may differ between storage vendors.  In some cases, applying QoS on datastore A may also impact datastore B due to storage array side NVMe over TCP QoS architecture and application.

This may result in:

• IO throttling seen on non-targeted datastore B.
• Higher than expected DAVG latency in esxtop for non-targeted datastore B.
• Performance has deteriorated messages in vmkernel.log for non-targeted datastore B.
  • 2026-05-06T21:02:01.269Z Wa(180) vmkwarning: cpu42:#)WARNING: StorageDeviceIO: 201: Device uuid.#### performance has deteriorated. I/O latency increased from average value of 4637 microseconds to 96766 microseconds.

Environment

ESXi 7.0U3+

NVMe over TCP storage that offer QoS solutions (such as Netapp)

Cause

Depending on the storage array vendor, NVMe over TCP traffic may use shared queues on the array side thus QoS throttling may consequently impact a non-targeted datastore.

Resolution

Engagement of NVMe over TCP storage vendor is recommended so the following items can be reviewed:

• QoS capabilities for NVMe over TCP traffic
• Potential NVMe over TCP QoS limitations due to storage connection architecture