Impact of NFS Latency on vCenter and VM Manageability

Products

• VMware vSphere ESXi

Issue/Introduction

Computation of moving average latency are added for NFSv3 from ESXi7.0 and NFSv4.1 from ESXi7.0 Update 1. The following log entries related to performance deterioration and improvement, and can be observed in the /var/run/log/vmkernel.log, if there is changes in the latency in the environment.

• When there is a performance deterioration, you will see similar to:

YYYY-MM-DDTHH:MM:SS cpu1:1000211299)WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> performance has deteriorated. I/O latency increased from average value of 4873(us) to 8900(us).

Above message is observed if the current moving average latency increases when compared to the average latency in the environment.

YYYY-MM-DDTHH:MM:SS cpu1:1000211299)WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> performance has deteriorated. I/O latency increased from average value of 16857(us) to 68722(us). Exceeded threshold 10000(us)

The above message is observed if the current moving average latency exceeds the threshold value (By default its 10ms).

YYYY-MM-DDTHH:MM:SS cpu1:1000211299)WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> average I/O latency 17269(us) has exceeded threshold 10000(us) for last 5 minutes

Above message is observed if the current moving average latency remains above the threshold when compared to the average latency in the environment.

• When there is a performance improvement:

YYYY-MM-DDTHH:MM:SS cpu23:1001391866)NFS: NFSVolumeLatencyUpdate:4590: NFS volume <NFS Volume name> performance has improved. I/O latency reduced from 68945(us) to 35389(us).

Above message is logged when there is improvement in the latency.

• System-wide performance degradation is observed due to high I/O latency on multiple NFS datastores. This issue manifests as unresponsive VMs including vCenter

Note: Similar logs will be observed in vmkwarning.log and vobd.log and also with NFSv4.1 volumes.

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

These logs indicate that the latency is occurring outside of ESXi. The root cause is a significant latency condition originating on the external NFS storage target or its network path.

Resolution

To resolve this issue investigate the environment outside of ESXi for the source of the latency. Troubleshooting should focus on the network path to the NFS target and the performance of the storage array itself, as the ESXi logs confirm the issue is external to the host.

Additional Information

Below information is from NetApp

NetApp have a feature on ONTAP storage called Elastic Sizing which seem to have stopped working during the issue.
NetApp KB on Elastic Sizing

Elastic Sizing with FlexGroup Volumes