Multiple virtual machines experience latency issues on local datastore.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms

• Application team reports latency on virtual machines residing on local datastore.

Environment

• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x

Cause

• SCSI Reserve/Release operation (SCSI Reservation > Perform I/O > SCSI Release) taking longer time than expected to complete. 
• Additionally, high DAVG (Device Average Latency) values report on controller and the storage devices.

Cause Validation

• The /var/run/log/vmkernel.log file reports "Long VMFS rsv time" errors. These error messages indicate that the SCSI Reserve/Release operation (SCSI Reservation > Perform I/O > SCSI Release) taking longer time than expected to complete.
  YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu20:2100394)FS3Misc: 1784: Long VMFS rsv time on 'Datastore_Name' (held for 401 msecs). # R: 1, # W: 1 bytesXfer: 24 sectors
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu4:2100438)FS3Misc: 1784: Long VMFS rsv time on 'Datastore_Name' (held for 363 msecs). # R: 1, # W: 1 bytesXfer: 24 sectors
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu37:2100434)FS3Misc: 1784: Long VMFS rsv time on 'Datastore_Name' (held for 504 msecs). # R: 1, # W: 1 bytesXfer: 24 sectors


• Drivers report as running on compatible versions.
  To verify the driver firmware version of the storage adapter, refer to the steps documented in Determining Network/Storage firmware and driver version in ESXi
  To verify driver compatibility refer Broadcom Compatibility guide
  
  
• High DAVG values reports across both the storage devices and controller. To review ESXTOP data, refer Using esxtop to identify storage performance issues for ESXi (multiple versions)
  
  • Storage Device stats:
    
    From the above screenshot, it was confirmed that storage device utilization reached 100%, indicating excessive I/O queuing. 
  • Adapter Stats:
    

Resolution

Engage your hardware vendor to further check the health of the storage device and the adapter.