QEDF driver I/O aborts on the host are causing the VM guest OS to reboot
search cancel

QEDF driver I/O aborts on the host are causing the VM guest OS to reboot

book

Article ID: 446200

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware Telco Cloud Infrastructure

Issue/Introduction

Virtual machines experience unexpected guest OS panics and repeated reboots. The underlying storage subsystem encounters stuck or timed-out storage I/O on the qedf fibre channel over ethernet (FCoE) initiator, forcing the driver to execute recovery resets and command aborts.

Symptoms are verified by the following signatures within vmkernel.log:

####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_internalAbortIO:####:Info: abort/virtual reset completed successfully, ref is: 3
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_eh_virtual_reset:####:Info: Returning: Success IOs aborted: 39, port_id[######]
####-##-##T##:##:##.###Z cpu##:#######)ScsiDeviceIO: ####: Cmd(0x45d941bc26c8) 0x2a, CmdSN 0x800e0006 from world ####### to dev "naa.#############################" failed H:0x7 D:0x0 P:0x0
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_eh_virtual_reset:####:Info: Returning: Success IOs aborted: 0, port_id[#####]
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_getCmdSatisfyingACondition:####:Info: Found the I/O for MatchingWorldID 0x4311d66e5fe0:0x1d6, state=1, ref=2
####-##-##T##:##:##.###Z cpu4:#######)qedf:vmhba##:qedfc_eh_virtual_reset:####:Info: vr: aborting oxid = 0x1d6, refcnt = 2, LBA = 2db88, cmdSN=0, worldId= #######
####-##-##T##:##:##.###Z cpu#:#######)ql_vmk_wait_for_completion:####:: Going to sleep.
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_fp_process_cqes:####:Info: dummy cqe. xid: 0x1d6
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_fp_process_cqes:####:Info: Abort cqe. xid: 0x1d6
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_process_abts_compl:####:Info: ABTS response - ACC Send RRQ
####-##-##T##:##:##.###Z cpu#:#######)ql_vmk_wait_for_completion:####:: Wokeup: status=Success.
####-##-##T##:##:##.###Z cpu#:#######)ql_vmk_wait_for_completion:####:: Returning VMK_OK
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_process_abts_compl:####:Info: (0:1): Completing cmd with Host Error status (0x7), xid=0x1d6, SN=#######, worldId=######, refcnt=3 lba=0x2db88 lbc=0x8 cmd ##:#:#:#:##
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_internalAbortIO:####:Info: abort/virtual reset completed successfully, ref is: 4
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_eh_virtual_reset:####:Info: Returning: Success IOs aborted: 1, port_id[######]
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_eh_virtual_reset:####:Info: Returning: Success IOs aborted: 0, port_id[######]
####-##-##T##:##:##.###Z cpu#:#######)ScsiDeviceIO: ####: Cmd(0x45b95ecfba48) 0x2a, CmdSN 0x3ce41b from world ####### to dev "naa.#############################" failed H:0x7 D:0x0 P:0x0
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_getCmdSatisfyingACondition:####:Info: Found the I/O for MatchingWorldID 0x4311d6807db0:0x53f, state=1, ref=2
####-##-##T##:##:##.###Z cpu4:#######)qedf:vmhba##:qedfc_eh_virtual_reset:####:Info: vr: aborting oxid = 0x53f, refcnt = 2, LBA = #####, cmdSN=0, worldId=#######
####-##-##T##:##:##.###Z cpu#:#######)ql_vmk_wait_for_completion:####:: Going to sleep.
####-##-##T##:##:##.###Z cpu##:#######)J#: ####: FS3_FSSyncIO failed on vol ###-####-## with IO was aborted by VMFS via a virt-reset on the device
####-##-##T##:##:##.###Z cpu##:#######)HBX: ####: '###-####-##': HB at offset ####### - Waiting for timed out HB:
####-##-##T##:##:##.###Z cpu##:#######)  [HB state abcdef02 offset ####### gen ## stampUS ################ uuid #######-########-####-################ jrnl <FB #######> drv ##.## lockImpl # ip ##.##.##.#]
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_fp_process_cqes:####:Info: dummy cqe. xid: 0x53f
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_fp_process_cqes:####:Info: Abort cqe. xid: 0x53f
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_process_abts_compl:####:Info: ABTS response - ACC Send RRQ
####-##-##T##:##:##.###Z cpu#:#######)ql_vmk_wait_for_completion:####:: Wokeup: status=Success.
####-##-##T##:##:##.###Z cpu#:#######)ql_vmk_wait_for_completion:####:: Returning VMK_OK
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_process_abts_compl:####:Info: (0:1): Completing cmd with Host Error status (0x7), xid=0x53f, SN=#######, worldId=#######, refcnt=3 lba=0x42458 lbc=0x8 cmd ##:#:#:#:##
####-##-##T##:##:##.###Z cpu#:#######)qedf:vmhba##:qedfc_internalAbortIO:####:Info: abort/virtual reset completed successfully, ref is: 3

Environment

ESXi: 7.0 U3w

TCI: 2.2

Cause

The primary cause is fabric-level link instability or dropped frames inducing link-level dropouts, which forces the qedf driver to initiate recovery resets and commands to fail with host transport error status H:0x7. Fiber Channel over ethernet (FCOE) and fiber channel (FC) host bus adapters are highly sensitive to driver and firmware interoperability, the synthetic host-side I/O resets and ABTS responses are directly correlated to deviations within the host bus adapter stack.

Understanding SCSI host-side NMP errors/conditions in ESXi

  1. SCSI Command Aborts: The vmkernel logs from host show explicit I/O failures characterized by NMP status H:0x7 D:0x0 P:0x0. This indicates that the local storage initiator driver (qedf) timed out waiting for a response from the storage processor and was forced to abort the active I/O exchanges (qedfc_eh_abort).

  2. RRQ Loop Event Signature: The sixty-five (65) Reinstate Recovery Qualifier (RRQ) events captured on vmhba## show the initiator actively attempting to flush the transport context. This pattern indicates protocol-layer or driver-level queuing exhaustion under burst workloads, independent of physical link health.

  3. Virtual Machine Impact: Due to the resulting path doubt state (nmp_DeviceRequestFastDeviceProbe), the VMFS clustering heartbeats to datastore ###-####-## were cancelled. This disk starvation caused the guest OS within VM to experience a prolonged block-device timeout, triggering a hard reset via the virtual chipset watchdog.

 

Resolution

  1. Isolate the scope of the failures by verifying if the command aborts and resets are limited to a single physical HBA (vmhba##) or specific active paths.

  2. Coordinate with the fabric and storage infrastructure vendors to perform a detailed health audit of the physical network layer, verifying switch stability, SFP integrity, and frame drops.

  3. Review and align the physical HBA firmware and driver versions with the officially validated combinations listed on the VMware Compatibility Guide.

Powered by Wolken Software