Users experiencing latency on ESXi host despite low disk IOPs

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms

• Users report application latency on virtual machines hosted on the ESXi host.

• All virtual machines are located on a single datastore.

• The datastore is backed by direct-attached storage (DAS). 

• The driver and firmware of the storage adapter in use is not listed in the Broadcom Compatibility guide

  To verify the driver firmware version of the storage adapter, refer to the steps documented in Determining Network/Storage firmware and driver version in ESXi

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

The high latency observed despite low disk IOPs is due to elevated DAVG values on the storage adapter as seen in ESXTOP. This indicates potential issues with I/O handling by the HBA. 

Cause Validation

Examination of the /var/log/vmkernel.log reveals recurring messages such as "Long VMFS rsv time". These messages indicate that the VMFS reservation time is unusually long, which can occur after a faulty firmware update on the storage controller or when the storage array is slow to respond due to internal issues.

2025-05-05T04:46:45.176Z cpu64:13709532) FS3Misc: 1755: Long VMFS rsv time on 'datastore' (held for 229 msecs) . # R: 1, # W: 1 bytesXfer: 24 sectors
2025-05-05T04:47:16.328Z cpu73:13709568) FS3M1SC: 1755: Long VMFS rsv time on 'datastore' (held for 223 msecs) . # R: 1, # W: 1 bytesXfer: 24 sectors

On validating the performance stats using the esxtop data, the IOPS values appear normal and under control. There is no latency observed on the storage device. The utilization of the device is less than 30% and the DAVG and KAVG values are under control.

However, on the storage adapter (vmhba), we see the DAVG values crossing 1000ms indicating an issue with the controller. 

Resolution

Engage your hardware vendor to further check the health of the storage adapter. Also validate the current firmware version of the adapter and ensure it aligns with the vendor’s compatibility and stability guidelines.