Storage Connectivity Disruptions during SAN Failover with Emulex HBAs

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi hosts may experience unexpected storage connectivity loss (All Paths Down - APD) and virtual machine unavailability during planned SAN node failovers, firmware upgrades, or Fibre Channel (FC) link resets. While some cluster nodes maintain access, specific hosts lose datastore connectivity due to non-compliant Emulex lpfc HBA driver/firmware combinations entering an unresponsive path state during storage transition events.

Symptoms

• vCenter Alerting: Datastores enter an All Paths Down (APD) state, causing impacted Virtual Machines to become unresponsive or unreachable.

• Path State Discrepancy: esxcli storage core path list shows paths as Active, but I/O repeatedly fails across those same paths.

• Host Log Signatures (vmkernel.log):

  • APD and Heartbeat errors: StorageApdHandler: 1191: APD start for [naa.ID] and esx.problem.vmfs.heartbeat.timedout.

  • SCSI status failures: H:0x1 (Host No Connect), H:0x5 (SCSI Timeout), or H:0x8 (Host Reset).

  • Driver abort storms: lpfc_abort_fcp_cmpl events during host maintenance.

  • Peer path loss and probe failures: Link Down Event x4 on vmhbaX followed by SATP Page 83 probes (No connection).

• Storage Array Signatures (e.g., IBM FlashSystem): Array logs report Internal Abort, Hung Xfer, or Forced Logout errors during SCSI_Write10 or SCSI_Write16 execution.

Environment

ESXi 8.x

Emulex HBAs

Cause

This issue is typically caused by a non-compliant HBA driver/firmware mismatch. Outdated Emulex HBA firmware is often incompatible with the currently installed lpfc driver baseline. This mismatch prevents the adapter from correctly processing Extended Link Services (ELS) requests and Fibre Channel link resets, resulting in the adapter hanging during maintenance events.

Resolution

1. Verify Compatibility: Review your specific HBA driver and firmware versions against the official Broadcom compatibility matrices (see References).
2. Align Firmware/Driver Versions: Flash the Emulex HBA firmware to the minimum required version or higher as specified by the installed lpfc driver baseline.
3. Validate: Perform a rolling restart of the ESXi hosts to ensure the storage stack re-initializes correctly. Verify path health via esxcli storage core path list.

Additional Information

• Broadcom KB 432997 ESXi Host Loses Fiber Channel Datastores Due to Emulex HBA Silent Driver Hang