Interpreting NFS datastore's latencies related messages in vmkernel logs

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
Computation of moving average latency are added for NFSv3 from ESXi7.0 and NFSv4.1 from ESXi7.0 Update 1. The following log entries related to performance deterioration and improvement, and can be observed in the vmkernel logs, if there is changes in the latency in the environment.

• When there is a performance deterioration, you will see similar to:

2020-08-17T00:05:28.270Z cpu1:1000211299)WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume NFS_DEV_100GB_1 performance has deteriorated. I/O latency increased from average value of 4873(us) to 8900(us).

Above message is observed if the current moving average latency increases when compared to the average latency in the environment.

2020-08-17T00:05:28.270Z cpu1:1000211299)WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume NFS_DEV_100GB_1 performance has deteriorated. I/O latency increased from average value of 16857(us) to 68722(us). Exceeded threshold 10000(us)

The above message is observed if the current moving average latency exceeds the threshold value (By default its 10ms).

2020-08-17T00:05:28.270Z cpu1:1000211299)WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume NFS_DEV_100GB_1 average I/O latency 17269(us) has exceeded threshold 10000(us) for last 5 minutes

Above message is observed if the current moving average latency remains above the threshold when compared to the average latency in the environment.

• When there is a performance improvement:

2020-06-08T05:58:52.455Z cpu23:1001391866)NFS: NFSVolumeLatencyUpdate:4590: NFS volume NFS_DEV_100GB_1 performance has improved. I/O latency reduced from 68945(us) to 35389(us).

Above message is logged when there is improvement in the latency.

Note: Similar logs will be observed in vmkwarning.log and vobd.log and also with NFSv4.1 volumes.

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

These log entries appear due to latency in the environment, outside of ESXi's NFS client.

Resolution

To resolve this issue investigate the environment outside of ESXi for the source of the latency.