Interpreting NFS datastore's latencies related messages in vmkernel logs

Products

• VMware vSphere ESXi

Issue/Introduction

Computation of moving average latency are added for NFSv3 from ESXi7.0 and NFSv4.1 from ESXi7.0 Update 1. The following log entries related to performance deterioration and improvement, and can be observed in the /var/run/log/vmkernel.log, if there is changes in the latency in the environment.

• When there is a performance deterioration, you will see similar to:

YYYY-MM-DDTHH:MM:SS cpu1:1000211299)WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> performance has deteriorated. I/O latency increased from average value of 4873(us) to 8900(us).

Above message is observed if the current moving average latency increases when compared to the average latency in the environment.

YYYY-MM-DDTHH:MM:SS cpu1:1000211299)WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> performance has deteriorated. I/O latency increased from average value of 16857(us) to 68722(us). Exceeded threshold 10000(us)

The above message is observed if the current moving average latency exceeds the threshold value (By default its 10ms).

YYYY-MM-DDTHH:MM:SS cpu1:1000211299)WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> average I/O latency 17269(us) has exceeded threshold 10000(us) for last 5 minutes

Above message is observed if the current moving average latency remains above the threshold when compared to the average latency in the environment.

• When there is a performance improvement:

YYYY-MM-DDTHH:MM:SS cpu23:1001391866)NFS: NFSVolumeLatencyUpdate:4590: NFS volume <NFS Volume name> performance has improved. I/O latency reduced from 68945(us) to 35389(us).

Above message is logged when there is improvement in the latency.

Note: Similar logs will be observed in vmkwarning.log and vobd.log and also with NFSv4.1 volumes.

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

These log entries appear due to latency in the environment, outside of ESXi's NFS client.

Resolution

To resolve this issue investigate the environment outside of ESXi for the source of the latency.

Additional Information

Below information is from NetApp

NetApp have a feature on ONTAP storage called Elastic Sizing which seem to have stopped working during the issue.
NetApp KB on Elastic Sizing

Elastic Sizing with FlexGroup Volumes