VSAN Disk Latency when using Intel E810c Network Interfaces

Products

• VMware vSAN 8.x

Issue/Introduction

- Optane p5800x drives cause brief poor performance/latency spikes when used as cache disks in vSAN where Intel E810c Network cards are used (Intel(R) Ethernet Controller E810-C for QSFP)

- Cache disk operations over 100MS latency under load.

- When using disk query, latency is evident:

vsish -e get /vmkModules/vmknvme/controllers//stats' in ESXi80u2, refer to the following cmds to get the controller number

1. esxcfg-scsidevs -A

2. esxcli nvme controller list > /vmkModules/vmknvme/controllers/260/> get /vmkModules/vmknvme/controllers/260/stats

Environment

- VSAN 8.0 U2

- Issue does not occur using other network interfaces, only Intel E810c

- Using AMD servers

- The BIOS setting in the “Dynamic Link Width Management (DWLM)” function seems to introduce the problem, with it set to “Forced” the extreme latency or “Receive Packet Dropped” dissipate.

 

Cause

- NVMe disk latency caused by combination of AMD BIOS settings and Intel E810c Ethernet Cards 

Resolution

 

Set BIOS settings on AMD chipset motherboard in accordance with third party documentation: Tuning Guide AMD EPYC 7003, section 2.1.2.