Cisco NFNIC HBA stays in "link-down" state after physical link is brought back up following Cisco Fabric Interconnect (FI) firmware upgrade

Products

• VMware vSphere ESXi
• VMware vSphere ESX 5.x
• VMware vSphere ESX 6.x
• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x
• VMware vSphere ESXi 5.0
• VMware vSphere ESXi 5.5
• VMware vSphere ESXi 8.0

Issue/Introduction

An Administrator will observe the link state for a HBA(s) remain in a "link-down" state instead of "link-up" following a firmware update to the Cisco Fabric Interconnect (FI):

$ localcli storage core adapter list
HBA Name  Driver    Link State  UID                                   Capabilities         Description
--------  --------  ----------  ------------------------------------  -------------------  -----------
vmhba0    vmw_ahci  link-n/a    sata.vmhba0                                                (0000:00:11.5) Intel Corporation Lewisburg SATA AHCI Controller
vmhba1    nfnic     link-up     fc.20############09:20############09  Second Level Lun ID  (0000:62:00.6) Cisco Corporation Cisco UCS VIC Fnic Controller
vmhba2    nfnic     link-down   fc.20############09:20############09  Second Level Lun ID  (0000:62:00.7) Cisco Corporation Cisco UCS VIC Fnic Controller
vmhba3    vmw_ahci  link-n/a    sata.vmhba3                                                (0000:04:00.0) Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller
vmhba32   vmkusb    link-n/a    usb.vmhba32                                                () USB

The Cisco Fabric Interconnect (FI) management interface will show that the physical link to that HBA is actually in a "link-up" state.

Cause

Upon reviewing the host information, particularly the output from the smbiosDump command, it becomes apparent that the Cisco UCS firmware version is incompatible with the Cisco NFNIC version:

SMBIOS version 3.2
  BIOS Info (Type 0): #0
    Size: 0x0001a
    Vendor: "Cisco Systems, Inc."
    Version: "B200M5.4.2.2d.0.0808221350" <--- This translates to Cisco UCS software version 4.2(2d)
    Date: "08/08/2022"

Cisco NFNIC driver version is 5.0.0.45:

$ localcli software vib list

Name                           Version                               Vendor  Acceptance Level  Install Date  Platforms
-----------------------------  ------------------------------------  ------  ----------------  ------------  ---------
nenic-ens                      1.0.9.0-1OEM.800.1.0.20613240         CIS     VMwareCertified   2024-10-30    host
nenic                          2.0.15.0-1OEM.800.1.0.20613240        CIS     VMwareCertified   2025-03-24    host
nfnic                          5.0.0.45-1OEM.800.1.0.20613240        CIS     VMwareCertified   2025-03-24    host

Resolution

Cisco has strictly enforced IO Device Driver to UCS Software/Firmware compatibility available in their UCS Interoperability Matrix. When these are out of sync, this is an unsupported configuration. To understand which UCS software/firmware is supported with which Cisco NFNIC driver, please refer to Cisco Compatibility Matrix: Cisco UCS Interoperability Matrix