Troubleshooting slow Virtual Disk performance on NFS Datastores

Products

• VMware vSphere ESXi

Issue/Introduction

Virtual machines running on NFS datastores may experience high latency due to external disconnections or timeouts. The following log entries relate to performance deterioration and improvement and can be observed in the vmkernel.log if there are changes in the latency in the environment.

• During performance deterioration, similar messages are logged in /var/run/log/vmkernel.log:


  • This message is observed if the current moving average latency increases when compared to the average latency in the environment.
    
    • WARNING: NFS: NFSVolumeLatencyUpdate:####: NFS volume performance has deteriorated. I/O latency increased from average value of 4873(us) to 8900(us)
      
      
  • This message is observed if the current moving average latency exceeds the threshold value (by default, it is 10 ms).
    
    • WARNING: NFS: NFSVolumeLatencyUpdate:####: NFS volume performance has deteriorated. I/O latency increased from average value of 16857(us) to 68722(us). Exceeded threshold 10000(us)
      
      
  • This message is observed if the current moving average latency remains above the threshold when compared to the average latency in the environment.
    • WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume average I/O latency 17269(us) has exceeded threshold 10000(us) for last 5 minutes

• • The following message occurs when there is improvement in the device latency: NFS: NFSVolumeLatencyUpdate:4590: NFS volume <NFS Volume name> performance has improved. I/O latency reduced from 68945(us) to 35389(us).

• A spike may appear in ARIA Operations:
  
  

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

These log entries appear due to high latency in the environment, outside of ESXi's NFS client.

Resolution

• Check with the storage vendor for vendor-specific NFS settings required on the ESXi host to improve performance.
• Additionally, investigate the external environment outside of ESXi to identify the source of the latency, and confirm with the vendor whether any backend storage rebuild or maintenance activity is occurring on the NFS storage system.

Additional Information

Interpreting NFS datastore's latencies related messages in vmkernel logs

An ongoing backend storage rebuild activity directly impacts NFS datastore performance. As a result, applications running on the guest operating systems experience degraded performance.