There is 30-second delay between the NVMe device reporting an error and the NVMe driver initiating a reset

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

In some cases, there is 30-second delay between the NVMe asynchronous event and the NVMe driver initiating a reset.

* In the NVMe asynchronous event handler, it takes up to 30 seconds to wait for all log page commands to complete.
* All log page commands time out, which usually indicates a hardware issue and the NVMe driver initiates a controller reset.

The wait time looks like a delay in response, but this is expected.

The log should like follow:
--Explicit error message from device came at 11:16:03:
YYYY-MM-DDT11:16:03.220Z Wa(180) vmkwarning: cpu34:#######)WARNING: NVMEDEV:9036 Failed to read log page 0x1, retry 0, controller 257.

--Failed to read NVMe event followed by controller reset event:
YYYY-MM-DDT11:16:23.222Z Wa(180) vmkwarning: cpu34:#######)WARNING: NVMEDEV:9036 Failed to read log page 0x1, retry 2, controller 257.
YYYY-MM-DDT11:16:23.222Z In(182) vmkernel: cpu34:#######)NVMEDEV:8245 Resetting controller 257 (nqn.1994-11.com.vendor:nvme:######:2.5-inch:##############)
YYYY-MM-DDT11:16:23.222Z In(182) vmkernel: cpu34:#######)NVMEDEV:8260 Controller 257 state changed from 5 to 8(INRESET)

Environment

VMware ESXi 8.0

Resolution

This issue is resolved in the VMware ESXi 8.0 Update 3j (Build 25429389).
The fix initiates the reset if any command times out so that the delay is reduced to 10 seconds.

Additional Information

VMware ESXi 8.0 Update 3j Release Notes
https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/release-notes/esxi-update-and-patch-release-notes/vsphere-esxi-80u3j-release-notes.html