NFS datastore latency caused by Intel icen driver RSS engine halts on ESXi 8.0U3

Products

• VMware vSphere ESXi

Issue/Introduction

• NFSv3 datastores performance is degraded, showing elevated disk latencies of 200–400 ms on the affected volumes.

• Multiple VMs are affected and latency is experienced mostly during database backup.

• From the /var/run/log/vmkernel.log, below performance impact is reported,

  vmkernel.log reports: WARNING: NFS: 5927: NFS volume [Name] performance has deteriorated. I/O latency increased from average value of 2023(us) to 296279(us)
vmkernel: cpu1:2097453)NetqueueBal: 4453: vmnic4: Cleaned up a NetQ RSS engine, 0 left

• Logs show frequent NetqueueBal: New NetQ RSS engine created followed by Cleaned up a NetQ RSS engine.

Environment

VMware ESXi 8.x

Cause

When using icen driver with Intel datastores run into performance bottleneck because of RSS feature is not enabled by default. The default configuration of disabled RSS is due to the consideration to reduce memory footprint, as RSS queues will reserve more memory.

Resolution

Manually change the module parameter:

esxcli system module parameters set -m icen -p "RSS=1,1,1... DRSS=4,4,4..."

Note : A mismatch between the icen driver and NIC firmware causes the Intel adapter to halt briefly while processing dynamic network load balancing requests (NetQueue/RSS). As NFS datastores rely on standard TCP/IP connectivity, these pauses result in severe I/O latency spikes.

Upgrade the firmware/driver version for Intel NICs, Refer article for determining network/storage firmware and driver version in ESXi - 323110

Additional Information

Enabling RSS and DRSS to improve performance on Intel icen driver