VMkernel logs report the message: Long VMFS3 rsv time
search cancel

VMkernel logs report the message: Long VMFS3 rsv time

book

Article ID: 345221

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
You see this message in the VMkernel logs:

FS3: 8496: Long VMFS3 rsv time on 'vm_tst2' (held for 419 msecs). # R: 1, # W: 1 bytesXfer: 5 sector


Environment

VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.5
VMware ESXi 4.1.x Installable
VMware ESX 4.0.x
VMware vSphere ESXi 5.0
VMware ESXi 4.0.x Embedded
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.1
VMware ESX 4.1.x

Cause

This issue occurs when a SCSI Reserve/Release operation (SCSI Reservation > Perform I/O > SCSI Release) takes longer than expected to complete. The expected reservation time is very fast, and if the reservation time exceeds 200ms, this message is reported. This is defined as the maximum reserve time or Scsi.MaxReserveTime.

Resolution

To resolve this issue, investigate latency on this storage device. Troubleshoot via esxtop as you would for a momentary performance issue. For more information, see Using esxtop to identify storage performance issues (1008205).

If these messages are seen for multiple LUNs around the same time, it indicates that the storage array is slow to process commands, most likely due to a higher than expected load. The time of day in which these messages are observed could indicate the cause for this high load:
  • If these message are observed overnight and on a nightly basis, this could indicate a higher load on the storage array due to backup operations.
  • If these messages are observed between 7am-9am and 4pm-6pm, and there is a virtual desktop infrastructure (VDI or View) on the storage array, this could indicate user load for logon/logoff operations.
  • If these messages are observed at the same time every day but fall outside the above two scenarios, it is likely related to a scheduled operation in the environment (such as a large SQL job or antivirus definition download for VDI).
  • If these messages are observed frequently but randomly, this could indicate that the storage is generally overloaded. Contacting the array vendor may be required to determine remediation steps (such as adding cache or moving from SATA drives to SCSI).
  • If these messages are observed rarely, it could point to future performance issues on the storage array.
To modify the trigger for this error message, increase the value of Scsi.MaxReserveTime from the default value to 500ms. This will not resolve latency concerns.

NOTE:The Atomic Test and Set (ATS) primitive is used for locking on Virtual Machine File System (VMFS) datastores for VMware vSphere Storage APIs for Array Integration (VAAI) compatible storage arrays. It is far superior to the SCSI Reservation locking technique. For more information on VAAI, see Frequently Asked Questions for vStorage APIs for Array Integration (1021976)
Verify if the storage array is supported for ATS VAAI primitive.
 
Caution: VMware recommends leaving this setting at the default value because the above messages indicate storage performance issues.
 
To increase or decrease the Scsi.MaxReserveTime value from the vSphere Client:
  1. Click the Configuration tab.
  2. Click Advanced  System Settings > Scsi > Scsi.MaxReserveTime.
  3. Set it to the desired value.
To make this same change from the command line, run this command:

In ESX/ESXi 4.x – esxcfg-advcfg -s <value> /Scsi/MaxReserveTime
In ESXi 5.x – esxcli system settings advanced set -i <value> -o /VMFS3/MaxReserveTime
 
Note: This value can be between 50ms and 500ms.


Additional Information

Powered by Wolken Software