Multiple VMs are not accessible due to high IOPS with the NFS datastore.

Products

• VMware vSphere ESXi

Issue/Introduction

• High IOPS observed on NFS datastore,resulted in all VMS residing in NFS datastore having access/performance issues.
• The following log entries related to performance deterioration and improvement, and can be observed in the /var/run/log/vmkernel.log

 2024-02-16T12:27:38.407Z cpu18:2098271)WARNING: NFS: 5015: NFS volume NFS_datastore performance has deteriorated. I/O latency increased from average value of 899(us) to 9654(us).
2024-02-16T12:27:41.479Z cpu10:2098272)WARNING: NFS: 5015: NFS volume NFS_datastore performance has deteriorated. I/O latency increased from average value of 1036(us) to 26207(us).
2024-02-16T12:29:28.173Z cpu21:2098272)NFS: 4971: NFS volume NFS_datastore performance has improved. I/O latency reduced from 26207(us) to 5170(us).
2024-02-16T12:30:03.900Z cpu22:2098272)WARNING: NFS: 5043: NFS volume NFS_datastore performance has deteriorated. I/O latency increased from average value of 5170(us) to 10279(us). Exceeded threshold 10000(us)
2024-02-16T12:30:04.145Z cpu17:2098272)NFS: 4971: NFS volume NFS_datastore performance has improved. I/O latency reduced from 9654(us) to 3683(us).
2024-02-16T12:30:04.549Z cpu7:2098271)WARNING: NFS: 5015: NFS volume NFS_datastore performance has deteriorated. I/O latency increased from average value of 1037(us) to 11347(us).
2024-02-16T12:31:16.668Z cpu16:2098272)WARNING: NFS: 5015: NFS volume NFS_datastore performance has deteriorated. I/O latency increased from average value of 1037(us) to 22843(us).
2024-02-16T12:32:27.519Z cpu9:2098272)NFS: 4971: NFS volume NFS_datastore performance has improved. I/O latency reduced from 22843(us) to 4564(us).

• Above message is observed if the current moving average latency remains above the threshold when compared to the average latency in the environment.

Environment

ESXi 7.x

ESXi 8.x 

Cause

These log entries appear due to latency in the environment, outside of ESXi's NFS client.

Resolution

To resolve this issue investigate the environment outside of ESXi for the source of the latency.