Intermittent VM Datastore Connectivity and VMK_SCSI_HOST_RESET Errors due to Faulty FC Fabric Interconnect Modules
search cancel

Intermittent VM Datastore Connectivity and VMK_SCSI_HOST_RESET Errors due to Faulty FC Fabric Interconnect Modules

book

Article ID: 450006

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms

  • Virtual machines experience intermittent connectivity issues with their associated datastores.

  • Applications running within the Guest OS may hang or report I/O errors.

  • Intermittent connectivity loss for specific VMs.

  • Significant command queuing observed at the device layer.

  • High frequency of SCSI host resets.

Environment

VMware vSphere ESXi

Cause

  • The issue is caused by faulty hardware interconnect modules within the Fibre Channel fabric. These modules fail to acknowledge SCSI commands within the required timeout window or lose the I/O sequence context. When the fabric fails to maintain context (evidenced by "Can't find context" and ABTS errors in switch logs), I/O commands remain unacknowledged. ESXi eventually exhausts the active command slot limit and begins queuing. To recover the path, the ESXi host initiates a VMK_SCSI_HOST_RESET (H:0x8) to clear the pending I/O, causing the observed intermittent connectivity.

  • In /var/run/log/vmkernel.log, below entries can be seen -

    Note: In below example vmhba 64 and 65 are converged adapter (eg: QLogic FastLinQ QL45xxx Series 25/50 GbE Controller FCoE) used for FC connectivity.

    • The device has 32 active commands executing, but 74 commands are queued up waiting. This proves the storage array/fabric is not acknowledging commands fast enough, causing a severe bottleneck.
      2026-07-22T18:14:18.256Z cpu48:2097335)ScsiDeviceIO: 4163: Cmd(0x45d947fe2e48) 0x89, cmdId.initiator=0x43088ee1c5c0 CmdSN 0x29fb2 from world 2097257 to dev "naa.################################" failed H:0x5 D:0x0 P:0x0 Cancelled from device layer.
      2026-07-22T18:14:18.256Z cpu48:2097335)Cmd count Active:32 Queued:74
      2026-07-22T18:14:18.256Z cpu69:2097763)qedf:vmhba64:qedfc_eh_abort:3010:Info: IO not found. Returning Success, cmdSN=29fb2, worldId=#######
      2026-07-22T18:14:18.256Z cpu69:2097763)qedf:vmhba65:qedfc_eh_abort:3010:Info: IO not found. Returning Success, cmdSN=29fb2, worldId=#######


      A standard write command (0x2a) is killed. The status H:0x8 means VMK_SCSI_HOST_RESET. The host has decided that the entire connection to the LUN needs to be reset, so all pending commands are instantly wiped out with an H:0x8 status. vmhba64 is communicating with port id 10800 on the SAN switch.
      2026-07-22T18:14:19.256Z cpu48:2097327)ScsiDeviceIO: 4096: Cmd(0x45d95518f188) 0x2a, cmdId.initiator=0x430cf2e12340 CmdSN 0xf6 from world ####### to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from device layer
      2026-07-22T18:14:19.256Z cpu61:2098382)qedf:vmhba64:qedfc_eh_virtual_reset:3141:Info: Returning: Success IOs aborted: 0, port_id[10800]
      2026-07-22T18:14:19.256Z cpu48:2097327)ScsiDeviceIO: 4096: Cmd(0x45d947fe5648) 0x2a, cmdId.initiator=0x430cf2e12340 CmdSN 0xf2 from world ####### to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from device layer
      2026-07-22T18:14:19.256Z cpu61:2098382)qedf:vmhba65:qedfc_getCmdSatisfyingACondition:2830:Info: Found the I/O for MatchingWorldID 0x43113e5f0de0:0x367, state=1, ref=3
      2026-07-22T18:14:19.256Z cpu48:2097327)ScsiDeviceIO: 4096: Cmd(0x45d955198288) 0x2a, cmdId.initiator=0x430cf2e12340 CmdSN 0xc2 from world ####### to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from device layer
      2026-07-22T18:14:19.256Z cpu61:2098382)qedf:vmhba65:qedfc_eh_virtual_reset:3112:Info: vr: aborting oxid = 0x367, refcnt = 3, LBA = 82736b0, cmdSN=0, worldId=2107968

      The driver waits, and the adapter hardware successfully acknowledges the reset and aborts the sequences (ABTS).
      2026-07-22T18:14:19.256Z cpu61:2098382)ql_vmk_wait_for_completion:1243 :: Going to sleep.
      2026-07-22T18:14:19.256Z cpu48:2097314)qedf:vmhba65:qedfc_fp_process_cqes:3506:Info: dummy cqe. xid: 0x367
      2026-07-22T18:14:19.256Z cpu48:2097314)qedf:vmhba65:qedfc_fp_process_cqes:3493:Info: Abort cqe. xid: 0x367
      2026-07-22T18:14:19.256Z cpu48:2097314)qedf:vmhba65:qedfc_process_abts_compl:1982:Info: ABTS response - ACC Send RRQ
      2026-07-22T18:14:19.256Z cpu65:2098382)ql_vmk_wait_for_completion:1248 :: Wokeup: status=Success.
      2026-07-22T18:14:19.256Z cpu65:2098382)ql_vmk_wait_for_completion:1269 :: Returning VMK_OK

      The LUN reset finishes successfully, temporarily clearing the blockage.
      2026-07-22T18:14:19.256Z cpu65:2098382)qedf:vmhba65:qedfc_internalAbortIO:2941:Info: abort/virtual reset completed successfully, ref is: 3
      2026-07-22T18:14:19.256Z cpu48:2097314)qedf:vmhba65:qedfc_process_abts_compl:2044:Info: (3:16): Completing cmd with Host Error status (0x7), xid=0x367, SN=c8, worldId=202a40, refcnt=3 lba=0x82736b0 lbc=0x8 cmd 2a:0:8:27:36



      Parallelly on the switch logs, fabric vendor will find below abort entries -

      $ showeventllog -d -debug -oneline -msg "ABTS"

      Error Example -"3:0:2 - ABTS from 0x10913 to 0x10c00. Can't find context. ox_id 0x57e rx_id 0xffff vp_queue_cdb N vpi 0x0"

Resolution

  • To resolve this issue, the physical integrity of the Fibre Channel path must be restored by replacing the failing fabric components.
    1. Engage Fabric/Hardware Vendor: Provide SAN switch and interconnect module logs to the hardware vendor for deep packet and error analysis.

    2. Identify Faulty Components: Review logs for ABTS (Abort Sequence) frames or messages indicating the switch "Can't find context." Pinpoint the specific ports or interconnect modules associated with these errors.

    3. Replace Hardware: Replace the identified faulty interconnect modules or cables within the fabric.

    4. Validate Connectivity: Monitor the ESXi logs to ensure the SCSI status codes H:0x2, H:0x5, and H:0x8 are no longer appearing.

      • Command to monitor logs:
        $ tail -f /var/run/log/vmkernel.log | grep -E "H:0x2|H:0x5|H:0x8"
Powered by Wolken Software