ESXi host unresponsive after nfnic HBA reset failure.

Products

VMware vSphere ESXi VMware vSphere ESX 8.x

Issue/Introduction

Following a fabric event (such as a Fiber Channel switch reboot), one or more Cisco HBAs may fail to recover, leading to host unresponsiveness.
Storage paths remain in a "Dead" state despite the physical link reporting as "Up."
One Cisco HBA (e.g., vmhba1) successfully recovers while the peer HBA (e.g., vmhba2) remains offline.
ESXi host becomes unresponsive (requires a hard reboot to recover).

Environment

VMware vSphere ESXi 8.x

Cause

The issue is triggered by a failure in the nfnic driver's recovery mechanism during a link-down event.
When a link-down event is detected, the driver attempts a firmware reset to clear the adapter state.
If this reset fails or times out (FNIC_IN_FW_RESET_TIMED_OUT_MODE) the adapter is left in an inconsistent IO_BLOCKED state.
Hence the storage stack cannot re-establish paths to the LUNs.

ESXi:/var/log/vmkernel.log
####-##-##T##:##:##.###Z In(182) vmkernel: cpu56:2098292)nfnic: <2>: INFO: fnic_handle_link: 1075: fnic2: recvd up to down event
####-##-##T##:##:##.###Z In(182) vmkernel: cpu56:2098292)nfnic: <2>: INFO: fnic_fdls_link_status_change: 98: fnic2: FDLS link status change link up:0, usefip:0
####-##-##T##:##:##.###Z In(182) vmkernel: cpu56:2098292)nfnic: <2>: INFO: fnic_fdls_link_down: 4584: Moving tport to PLOGO state tport: 7 fcid: 0x150080
####-##-##T##:##:##.###Z In(182) vmkernel: cpu56:2098292)nfnic: <2>: INFO: fnic_fdls_link_down: 4584: Moving tport to PLOGO state tport: 8 fcid: 0x150280
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu56:2098292)WARNING: nfnic: <2>: fnic_fw_reset_handler: 312: Issued fw reset
####-##-##T##:##:##.###Z In(182) vmkernel: cpu100:2098728)nfnic: <2>: INFO: fnic_queuecommand: 745: iport not in ready state. State: 1 or FNIC_IPORT_IO_BLOCKED
####-##-##T##:##:##.###Z In(182) vmkernel: cpu54:2098908)nfnic: <2>: INFO: fnic_queuecommand: 745: iport not in ready state. State: 1 or FNIC_IPORT_IO_BLOCKED.
.
####-##-##T##:##:##.###Z In(182) vmkernel: cpu63:2097960)nfnic: <2>: INFO: fnic_taskMgmt: 2146: VMK_SCSI_TASKMGMT_ABORT issued. iport not in ready state. state: 1 or FNIC_IPORT_IO_BLOCKED
####-##-##T##:##:##.###Z In(182) vmkernel: cpu56:2098292)nfnic: <2>: INFO: fnic_scsi_fcpio_reset: 5089: fw reset on fnic(2): FAILED fnic state: 1
####-##-##T##:##:##.###Z In(182) vmkernel: cpu56:2098292)nfnic: <2>: INFO: fnic_scsi_fcpio_reset: 5096: Setting fnic to FNIC_IN_FW_RESET_TIMED_OUT_MODE
####-##-##T##:##:##.###Z In(182) vmkernel: cpu56:2098292)nfnic: <2>: INFO: fnic_fdls_link_down: 4614: Deleting tport: 7, fcid: 0x150080 itemPtr: 0x4314cc81cee0

Resolution

Verify on the VMware Hardware Compatibility Guide that your Cisco adapter's current nfnic driver and VIC firmware are listed together as a certified pair for your specific ESXi version.
Determining Network/Storage firmware and driver version in ESXi
Engage your Hardware vendor to investigate driver logging and confirm the recommended driver and firmware as per Cisco.
Engage your SAN vendor to verify if the fabric switch is receiving and acknowledging FLOGI requests from the host during recovery events.