VMkernel logs report the message: Long VMFS rsv time

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi host becomes unresponsive.

 

In VMkernel logs following events seen:

FS3: 8496: Long VMFS3 rsv time on 'vm_tst2' (held for 419 msecs). # R: 1, # W: 1 bytesXfer: 5 sector

Cause

When a SCSI Reserve/Release operation (SCSI Reservation > Perform I/O > SCSI Release) takes longer than expected to complete. The expected reservation time is very fast, and if the reservation time exceeds 200ms, this message is reported. This is defined as the maximum reserve time or Scsi.MaxReserveTime.

 

This error may also be seen after a faulty firmware update on a storage controller.

Resolution

Investigate latency between the ESXi host and the storage device. Troubleshoot via esxtop to confirm performance issues.

Note: For more information, see Using esxtop to identify storage performance issues for ESXi (multiple versions).

If these messages are seen for multiple LUNs around the same time, it indicates that the storage array is slow to process commands, most likely due to a higher than expected load. The time of day or frequency in which these messages are observed could indicate the cause for the high load. See below for some examples.

• Overnight and on a nightly basis.
  • Could indicate a higher load on the storage array due to backup operations.
• Between 7am-9am and 4pm-6pm, and there is a virtual desktop infrastructure (VDI or View) on the storage array.
  • Could indicate user load for logon/logoff operations.
• At the same time every day but fall outside the above two scenarios.
  • It is likely related to a scheduled operation in the environment (such as a large SQL job or antivirus definition download for VDI).
• Frequent but random.
  • Could indicate that the storage is generally overloaded. Contacting the array vendor may be required to determine remediation steps (such as adding cache or moving from SATA drives to SCSI).
• Rarely.
  • Could point to future performance issues on the storage array.

The trigger for this error message can be increased by modifying the value for Scsi.MaxReserveTime from the default value to 500ms. However, this will not resolve the underlying issue. Please see more information on this below.

Note: The Atomic Test and Set (ATS) primitive is used for locking on Virtual Machine File System (VMFS) datastores for VMware vSphere Storage APIs for Array Integration (VAAI) compatible storage arrays. It is far superior to the SCSI Reservation locking technique. For more information on VAAI, see Frequently Asked Questions for vStorage APIs for Array Integration.

 

Verify if the storage array is supported for ATS VAAI primitive.

 

Caution: VMware recommends leaving this setting at the default value because the above messages indicate storage performance issues.

 

To increase or decrease the Scsi.MaxReserveTime value from the vSphere Client:

1. Select the ESXi host.
2. Click the Configuration tab.
3. Click Advanced  System Settings.
   • Take note of current value for Scsi.MaxReserveTime in case it is needed for future reference.
4. Click "Edit" and set Scsi.MaxReserveTime to the desired value. This value can be between 50ms and 500ms.

Note: If you need additional assistance with making these changes, please create a support case with our Storage team.