Troubleshooting ESXi Fibre Channel Dead Paths and H:0x1 SCSI Connection Errors

Products

VMware vSphere ESXi

Issue/Introduction

This article addresses intermittent and persistent Fibre Channel (FC) storage path failures occurring on VMware vSphere ESXi hosts.

The issue is characterized by recurring H:0x1 (NO_CONNECT) SCSI host status errors reported by the Native Multipathing (NMP) plugin, coupled with driver-level timeouts.

ESXi hosts reporting storage path failures (Dead Paths) on specific Fibre Channel HBAs (e.g., vmhba5).
VMkernel logs reporting H:0x1 D:0x0 P:0x0 SCSI status errors during I/O operations.
Driver-level warnings regarding device loss, including:
- lpfc_start_devloss
- lpfc_dev_loss_tmo_handler
- BLS ABORT RSP failed
- ELS cmd x19 error
Registered State Change Notification (RSCN) timeouts or discovery state machine delays.

Environment

VMware vSphere ESXi 8.X

Cause

The primary root cause is physical fabric-level instability or component degradation. The H:0x1 status code (VMK_SCSI_HOST_NO_CONNECT) indicates the host storage driver cannot establish communication with the target FC port. This is typically triggered by:

Faulty or degraded optical transceivers (SFPs).
Damaged or improperly seated fiber optic patch cables.
Port-level instability on the upstream SAN switch (e.g., CRC errors, signal loss).
Target array port unresponsiveness.

These physical issues lead to link oscillation, ELS command failures (e.g., PLOGI/ADISC), and RSCN timeouts, which force the lpfc driver to trigger 10-second device loss timeouts (devloss_tmo).

Resolution

To resolve this issue, perform a thorough physical and fabric-layer audit of the storage infrastructure associated with the impacted HBA.

Prerequisites

Ensure that any critical VMs are moved to alternate hosts if the storage path instability is causing I/O latency or datastore APD/PDL states.
Access to the SAN Switch management interface.

Steps:

Analyze logs for physical layer events Review /var/run/log/vmkwarning.log for hardware-specific errors. Look for messages indicating physical degradation:
```
WARNING: lpfc: lpfc_sli4_async_sli_evt: Port 2 :Optics faulted/incorrectly installed/not installed - Reseat optics, if issue not resolved, replace.
```
Audit physical components
- Reseat the optical transceivers (SFPs) on both the ESXi HBA port and the switch port.
- Replace the SFP transceivers and fiber patch cables to rule out component failure.
- Verify cable integrity (check for kinks or sharp bends).
Inspect SAN switch port health Log into the SAN switch (Brocade/Cisco) and audit the port statistics mapping to the host HBA and the storage target ports. Search for:
- CRC (Cyclic Redundancy Check) errors.
- Encoding errors (enc_out / disc_c).
- Tx/Rx power levels outside of expected thresholds.
Verify Driver and Firmware Compatibility Ensure the Emulex lpfc driver version and HBA firmware are on the current Broadcom/VMware Compatibility Guide (VCG) alignment matrix. Mismatched versions can exacerbate thread starvation during fabric rediscovery.
Force Path Re-evaluation After verifying physical connectivity, manually trigger an adapter rescan to allow the NMP stack to re-evaluate path availability:
```
esxcli storage core adapter rescan --adapter=vmhba#
```
(Replace # with the affected vmhba identifier, e.g., vmhba5)

Workaround If path instability continues to cause log flooding or I/O performance degradation, mask or disable the flapping paths temporarily at the SAN switch level to prevent the ESXi host from attempting I/O on unstable links.

Additional Information

For a comprehensive list of SCSI host status definitions, refer to Broadcom KB 309543.
Refer to the VMware Compatibility Guide to confirm the latest supported driver and firmware combinations for your specific HBA model and ESXi version.