The ESXi host experiencing latency and intermittent APDs on NFS datastores.

Products

• VMware vSphere ESXi
• VMware vSphere ESX 8.x
• VMware vSphere ESX 7.x

Issue/Introduction

• The VM's may go into a hung state/unresponsive.

• The VMs running on the ESXi hosts experience slowness and are unable to migrate to any host.

• Storage vmotion from one NFS share to another takes a long time.

• Performance has deteriorated messages for the NFS datastores observed in the /var/log/vmkernel.log file of the ESXi host:

            WARNING: NFS: NFSVolumeLatencyUpdate:4879: NFS volume <NFS Volume name> performance has deteriorated. I/O latency increased from average value of 16857(us) to 68722(us). Exceeded threshold 10000(us)

• Intermittent APD events reported for the NFS datastore in the /var/log/vmkernel.log file of the ESXi host:

            StorageApdHandlerEv: 110: Device or filesystem with identifier [########-########] has entered the All Paths Down state.

• Running esxtop on the ESXi through CLI reports High DAVG value indicating latency from the storage device (LUN):

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

 

Cause

• A temporary network or NFS server availability issue disrupted connectivity between the ESXi host and the NFS server.
• A faulty port on switch or faulty SFP was identified.

Resolution

• Reboot all ESXi hosts to clear any residual references for the affected datastores following an APD (All Paths Down).
  
  
• Engage both the storage and switch vendor to investigate on latency and APD incidents.

Additional Information

Please refer the below KB articles for NFS datastore's APD issues.

cancel ESXi hosts reports sudden NFS APD events where the host enters and exists an APD without any interaction.</a >

Permanent Device Loss (PDL) and All-Paths-Down (APD) on host</a >

Refer the KB article Troubleshooting NFS datastore connectivity issues</a > for latency on NFS datastore.