Impact of NFS Latency on vCenter and VM Manageability

Products

• VMware vSphere ESXi

Issue/Introduction

Computation of moving average latency are added for NFSv3 from ESXi7.0 and NFSv4.1 from ESXi7.0 Update 1. The following log entries related to performance deterioration and improvement, and can be observed in the /var/run/log/vmkernel.log, if there is changes in the latency in the environment.

• When there is a performance deterioration, you will see similar to:

YYYY-MM-DDTHH:MM:SS cpu1:1000211299)WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> performance has deteriorated. I/O latency increased from average value of 4873(us) to 8900(us).

Above message is observed if the current moving average latency increases when compared to the average latency in the environment.

YYYY-MM-DDTHH:MM:SS cpu1:1000211299)WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> performance has deteriorated. I/O latency increased from average value of 16857(us) to 68722(us). Exceeded threshold 10000(us)

The above message is observed if the current moving average latency exceeds the threshold value (By default its 10ms).

YYYY-MM-DDTHH:MM:SS cpu1:1000211299)WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> average I/O latency 17269(us) has exceeded threshold 10000(us) for last 5 minutes

Above message is observed if the current moving average latency remains above the threshold when compared to the average latency in the environment.

• When there is a performance improvement:

YYYY-MM-DDTHH:MM:SS cpu23:1001391866)NFS: NFSVolumeLatencyUpdate:4590: NFS volume <NFS Volume name> performance has improved. I/O latency reduced from 68945(us) to 35389(us).

Above message is logged when there is improvement in the latency.

• System-wide performance degradation is observed due to high I/O latency on multiple NFS datastores. This issue manifests in different ways, as for example:
  • Unresponsive VMs, which may include vCenter.
  • Slow or failed vMotion tasks.
  • Slow or failed Cloning tasks.

Note: Similar logs will be observed in vmkwarning.log and vobd.log and also with NFSv4.1 volumes.

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

These logs indicate that the latency is occurring outside of ESXi. The root cause is a significant latency condition originating on the external NFS storage target or its network path.

Resolution

To resolve this issue investigate the environment outside of ESXi for the source of the latency.

Troubleshooting should focus on the network path to the NFS target and the performance of the storage array itself, as the ESXi logs confirm the issue is external to the host.

Diagnostic Procedures:

Before concluding that NFS latency is caused by array saturation, perform the following physical layer health check to rule out underlying transport issues:

1. Monitor Latency Trends:

   • Open esxtop on the affected host.
   • Press n to switch to the NFS view.
   • Press f and toggle fields to ensure DAVG/cmd and KAVG/cmd are visible.
   • Note: If latency spikes correlate strictly with VM power-on/reboot events, investigate array-level contention. If latency spikes occur randomly or consistently during low I/O, focus on the physical network path.

2. Physical Layer Integrity Check:

   • Check the physical switch ports connected to the ESXi host uplinks and storage array ports.
   • Look for incrementing CRC errors, input errors, or runts.
   • Verify SFP transceiver diagnostics (if supported by the switch) for abnormal light levels or signal degradation.
   • Note: Even a single faulty SFP can cause periodic packet loss that mimics "datastore latency" in VMware logs.

Resolution & Workarounds (Updated)

If the diagnostics indicate high latency without a clear storage array bottleneck, apply the following steps:

1. Transport Remediation:

   • Replace suspect SFP transceivers or network cables on the path between the host and the storage array.
   • Ensure all physical switch ports are configured with PortFast (or equivalent edge-port configurations) to prevent traffic black-holing during link-state transitions or LACP/PortChannel negotiation.

2. Host-Side Tuning:

   • Verify that NIC driver versions (nfnic, icen, etc.) and firmware match the vendor's Hardware Compatibility Guide (HCG) requirements.
   • If utilizing NFS 3, ensure that the load-balancing policy on the virtual switch is set to "Route based on physical NIC load" (LBT) or "Route based on originating virtual port" to minimize path oscillation.

3. Advanced Verification:

   • Review vmkernel.log for WARNING: NFS: 5043: NFS volume <datastore> performance has deteriorated.
   • If these warnings occur, cross-reference the timestamps with switch logs to determine if a physical link event preceded the latency spike.