NFS poor read throughput when there is ESX packet loss: NFS server slow start interacts with ESX delayed ACK

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
ESX NFS read throughput for a storage array is extremely poor

Environment

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.7

Cause

• How to identify problem:
  • Network traffic was captured (via inline optical tap) between Esx and storage array, using hardware packet capture system.
  • Packet trace file was analyzed using Wireshark.
• TCP interaction: Slow start on the storage array interacts in an undesirable way with delayed ACK on ESX, leading to long (100ms) pauses in data transfer.
• When ESX delayed ACK was disabled, NFS read throughput increased significantly.

Note:See ESXi NFS Read Performance: TCP Interaction between Slow Start and Delayed Acknowledgement. for a detailed analysis

Resolution

This is a known issue affecting vSphere 6.7

Currently, there is no resolution.

Workaround:
For vSphere 7.0 and later:

ESX: Disable TCP Delayed ACK (100ms) for NFS (SunRPC)

• Disable: esxcli system settings advanced set -o "/SunRPC/SetNoDelayedAck" -i 1
• Verify: esxcli system settings advanced list | grep -A10 /SunRPC/SetNoDelayedAck