vHBA controller status shows link-down after Cisco Fabric Interconnect firmware update

Products

• VMware vSphere ESX 8.x

Issue/Introduction

• Following a firmware upgrade on one of the Cisco Fabric Interconnects (FIs), associated vHBA controllers are displaying a "link-down" status and are non-operational.
  
  
• esxcfg-scsidevs -a output confirms that vmhba1 and vmhba3 (connected to the upgraded FI) are in a link-down state, while others remain link-up.
  

  HBA Name  Driver    Link State         UID                                                        Capabilities                       Description
  --------          --------  ----------        ------------------------------------                     -------------------                -----------
  vmhba0    nfnic        link-up         fc.################################   Second Level Lun ID      (0000:1c:00.0) Cisco Corporation Cisco UCS VIC Fnic Controller
  vmhba1    nfnic        link-down   fc.################################   Second Level Lun ID      (0000:1c:00.1) Cisco Corporation Cisco UCS VIC Fnic Controller
  vmhba2    nfnic        link-up         fc.################################   Second Level Lun ID      (0000:1c:00.2) Cisco Corporation Cisco UCS VIC Fnic Controller
  vmhba3    nfnic        link-down   fc.################################   Second Level Lun ID      (0000:1c:00.3) Cisco Corporation Cisco UCS VIC Fnic Controller

• VMkernel logs indicate a sequence of events: a link status change, followed by a firmware crash, and subsequent 'iport not ready' messages.
  YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu23:######)nfnic: <2>: INFO: fnic_handle_link: 1001: link status 0 down cnt 1
  YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu23:######)nfnic: <2>: INFO: fnic_handle_link: 1003: old status 1 old down cnt 0
  YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu23:######)nfnic: <2>: INFO: fnic_handle_link: 1075: fnic2: recvd up to down event
  YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu23:######)nfnic: <2>: INFO: fnic_fdls_link_status_change: 98: fnic2: FDLS link status change link up:0, usefip:0
  YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu23:######)nfnic: <2>: INFO: fnic_fdls_link_down: 4584: Moving tport to PLOGO state tport: 1 fcid: 0x######
  YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu23:######)nfnic: <2>: INFO: fnic_fdls_link_down: 4584: Moving tport to PLOGO state tport: 2 fcid: 0x######
  YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu23:######)nfnic: <2>: INFO: fnic_fdls_link_down: 4584: Moving tport to PLOGO state tport: 3 fcid: 0x######
  YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu23:######)nfnic: <2>: INFO: fnic_fdls_link_down: 4584: Moving tport to PLOGO state tport: 4 fcid: 0x######
  YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu23:######)WARNING: nfnic: <2>: fnic_fw_reset_handler: 312: Issued fw reset
  YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu9:######)nfnic: <2>: INFO: fnic_queuecommand: 745: iport not in ready state. State: 1 or FNIC_IPORT_IO_BLOCKED
  YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu24:######)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x8a (0x45ba10de5480, 2101881) to dev "naa.#######################" on path "vmhba1:C#:T#:L#" Failed:
  YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu24:######)NMP: nmp_ThrottleLogForDevice:3898: H:0x1 D:0x0 P:0x0 . Act:FAILOVER. cmdId.initiator=0x430e76a39200 CmdSN 0x324
  
  

Environment

VMware vSphere ESXi 8.x

Resolution

Engage Cisco (as the hardware and firmware vendor for UCS/FI components and the fNIC driver) to investigate the root cause of the vHBA link failures and the subsequent inability to recover after the FI firmware upgrade.