Regular Virtual Machine I/O can overload 10G NICs in certain disk-dense vSAN OSA configurations and result in storage latency

Products

• VMware vSAN

Issue/Introduction

In certain NVMe - dense vSAN OSA configurations (multiple-NVMe capacity devices, 2+ diskgroups per host) Virtual Machine I/O alone can overload 10G network. 
This can be observed even without simultaneous/additional I/O operations added by resync/rebuild (as described in https://knowledge.broadcom.com/external/article/371138/)

This scenario can be confirmed by comparing 'Metrics for vSAN Host Network' (at Host > Monitor > vSAN / Performance > Host Network) against theoretical bandwidth of the connected uplinks. Example below is showing 10Gbps uplink fully saturated - network I/O throughput values seen at/around 1GB/s (note differing units (bit vs byte) used, for more information on data-rate unit conversion, see: https://en.wikipedia.org/wiki/Data-rate_units):

Such situations are often accompanied by: packet drops, retransmits and TCP congestion seen.

Environment

vSAN OSA 8.x, vSAN OSA 9.x

 

Cause

Storage traffic relies on consistent delivery across the network to ensure predictable performance. Non-deterministic latencies caused by the network contention will result in higher, unpredictable latency in vSAN. 

Resolution

Consider upgrading your networking infrastructure to 25Gb or higher (in case of very disk-dense vSAN environments).

Additional Information

For more information, see: Network Design for vSAN for VMware Cloud Foundation at https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/vcf-design-5-2/vcf-vsan-design.html#GUID-05D7B6D5-B737-444E-9983-9D7A274F9453-en