NFS Performance Bottleneck due overloaded single connection

Products

• VMware vSphere ESXi

Issue/Introduction

In /var/run/log/vmkernel.log we can see average latency (in us) spiking way higher than the 10000us default latency:

YYYY-MM-DD:HH:MM:SS Wa(180) vmkwarning: cpu1:2098344)WARNING: NFS: 5965: NFS volume #### performance has deteriorated. I/O latency increased from average value of 0(us) to 94962(us). Exceeded threshold 10000(us)
YYYY-MM-DD:HH:MM:SS Wa(180) vmkwarning: cpu44:2098344)WARNING: NFS: 5965: NFS volume #### performance has deteriorated. I/O latency increased from average value of 94962(us) to 192295(us). Exceeded threshold 10000(us)
YYYY-MM-DD:HH:MM:SS Wa(180) vmkwarning: cpu29:2098344)WARNING: NFS: 5965: NFS volume #### performance has deteriorated. I/O latency increased from average value of 192295(us) to 386468(us). Exceeded threshold 10000(us)

Other possible symptoms:

• Storage vMotion timeouts or high completion time
• Snapshot creation/consolidation
• Guest OS hung/slowness or crashes.
• Packet drops or duplicate ACK 

 

Environment

VMware vSphere ESXi 8.x

Cause

Workloads that may have higher throughput may overwhelm the singular TCP connection stream from the host to the NFS server. This bottleneck can lead to increased latency and contention.

Output of the command esxcli storage nfs list shows that there is only a singular TCP connection active between the ESXi host and the NFS server (Connections: 1)

Resolution

The amount of total TCP stream/connections can be increased using the nConnect feature to parallelize the workload. 

See more in; Support for nConnect feature added in ESXi NFS client.