VMs on NFS datastore observe latency

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Degraded performance has been observed within Guest OS operations for VMs residing on specific host and NFS datastore combination.

Environment

• VMware vSphere ESXi 7.x

• VMware vSphere ESXi 8.x

Cause

• The /var/run/log/vmkernel.log of ESXi host where the VM was running reports datastore latency issue:

  2026-01-12T07:09:56.025Z Wa(180) vmkwarning: cpu72:2099720)WARNING: NFS: 5927: NFS volume <nfs_datastore_name> performance has deteriorated. I/O latency increased from average value of 1091(us) to 11068(us).
2026-01-12T07:19:14.933Z Wa(180) vmkwarning: cpu97:2099720)WARNING: NFS: 5965: NFS volume <nfs_datastore_name> performance has deteriorated. I/O latency increased from average value of 0(us) to 10245(us). Exceeded threshold 10000(us)

• Additionally, IOs are queuing at the ESXi host level and exhausting the buffer. This bottleneck is triggered by high latency from the NFS server, which is failing to acknowledge processed IOs in a timely manner: (/var/run/log/vmkernel.log)

  2026-01-12T07:06:28.119Z In(182) vmkernel: cpu120:2099902)SunRPC: 5704: Socket space full. rpc queued. 0xc9c2c0b5.

• Comparing the "maxQDepthReached" counter for the NFS datastore on the host before and after a task in run within the VM's GuestOS, shows an increase.
  To view this value: SSH to the ESXi host where the VM resides and run the below vsish command:
  root@esxi# vsish
> cd /vmkModules/nfsclient/mnt/<nfs_datastore_name>/
> cat properties
mount point information {
  volume name:<nfs_datastore_name>
.
.
  maxQDepthReached:3456789
.
.

• The queueing of the NFS datastore IO commands on the ESXi host also causes the KAVG (Latency caused by VMKernel) to go high along with the DAVG (Latency at the device driver level) for the NFS datastore. This can be viewed by the esxtop command on the ESXi host. ESXTOP overview for Performance Troubleshooting.

• All of the above factors indicate NFS server latency.

Resolution

Engage the storage vendor to investigate the cause of NFS server latency.