Virtual Disks on NFS Datastores are Performing Slowly

Products

• VMware vSphere ESXi

Issue/Introduction

 When running a VM on NFS datastores you may experience high latency if there is external disconnections or timeouts. The following log entries related to performance deterioration and improvement, and can be observed in the vmkernel.log, if there is changes in the latency in the environment.

• When there is a performance deterioration, you will see similar in /var/run/log/vmkernel.log

WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> performance has deteriorated. I/O latency increased from average value of 4873(us) to 8900(us).

Above message is observed if the current moving average latency increases when compared to the average latency in the environment.

 

WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> performance has deteriorated. I/O latency increased from average value of 16857(us) to 68722(us). Exceeded threshold 10000(us)

The above message is observed if the current moving average latency exceeds the threshold value (By default its 10ms).

WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> average I/O latency 17269(us) has exceeded threshold 10000(us) for last 5 minutes

Above message is observed if the current moving average latency remains above the threshold when compared to the average latency in the environment.

• When there is a performance improvement:

NFS: NFSVolumeLatencyUpdate:4590: NFS volume <NFS Volume name> performance has improved. I/O latency reduced from 68945(us) to 35389(us).

Above message is logged when there is improvement in the device latency.

Environment

ESXi 7.0.X

ESXi 8.0.X

Cause

These log entries appear due to the high latency in the environment, outside of ESXi's NFS client.

 

Resolution

Please check with the storage vendor for any vendor specific NFS settings needed on ESXi host to improve performance or to resolve this issue investigate the environment outside of ESXi for the source of the latency. 

 

Additional Information

Please refer following kb for additional information 

Interpreting NFS datastore's latencies related messages in vmkernel logs