HBA(s) show "Link-down" status when failing to successfully perform PLOGI to fabric port

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 5.0
• VMware vSphere ESXi 5.5
• VMware vSphere ESXi 8.0
• VMware vSphere ESX 5.x
• VMware vSphere ESX 6.x
• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x

Issue/Introduction

An Administrator will observe HBA(s) in a "link-down" state when querying the status of the HBAs:

$ localcli storage core adapter list
HBA Name  Driver    Link State  UID                                   Capabilities         Description
--------  --------  ----------  ------------------------------------  -------------------  -----------
vmhba0    nfnic     link-down   fc.20############22:20############88  Second Level Lun ID  (0000:1b:00.3) Cisco Corporation Cisco UCS VIC Fnic Controller
vmhba1    nfnic     link-down   fc.20############22:20############89  Second Level Lun ID  (0000:1b:00.4) Cisco Corporation Cisco UCS VIC Fnic Controller
vmhba2    nfnic     link-up     fc.20############22:20############8a  Second Level Lun ID  (0000:d3:00.3) Cisco Corporation Cisco UCS VIC Fnic Controller
vmhba3    nfnic     link-up     fc.20############22:20############8b  Second Level Lun ID  (0000:d3:00.4) Cisco Corporation Cisco UCS VIC Fnic Controller
vmhba4    vmw_ahci  link-n/a    sata.vmhba4                                                (0000:50:00.0) Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller

Cause

When reviewing /var/log/vmkernel.log, after the HBA(s) successfully completed the FLOGI process, the PLOGI process starts but gets interrupted with Fabric Logout (LOGO) requests from a specific NPort: 

2025-04-24T17:56:24.506Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fdls_process_logo_req: 3645: Process LOGO request from fcid: 0xfffc53 <---- HBA being kicked off the fabric by NPORT 0xfffc53
2025-04-24T17:56:24.506Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fdls_process_logo_req: 3649: Received LOGO req from 0xfffc53 in iport state:3 .Dropping the frame.
2025-04-24T17:56:26.925Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5

The HBA driver attempts to retry the PLOGI attempts however it will hit a maximum retry and give up:

2025-04-24T17:56:47.099Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048 <---- HBA keeps sending PLOGIs over and over again without success
2025-04-24T17:56:51.101Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:57:11.102Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:57:15.104Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:57:35.105Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:57:39.107Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:57:59.109Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:58:03.110Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:58:23.112Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:58:27.114Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:58:47.115Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:58:51.117Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:59:11.118Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:59:15.120Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:59:35.122Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_error_fabric_disc: 2687: FDLS discovery error from 5 state <---- PLOGI failed and HBA driver gives up retrying

Resolution

Engage the SAN Switch vendor to troubleshoot further. Not only is a particular NPort ID is repeatedly asking the HBA to perform a fabric logout (LOGO) but, after many retries, the HBA is not able to successfully perform a PLOGI. These issues cannot be resolved from the ESXi side.

Additional Information

Here is a view of the Complete sequence:

First we have a link up for the adapter:

2025-04-24T17:56:22.199Z In(182) vmkernel: cpu46:2098180)nfnic: <1>: INFO: fnic_handle_link: 1001: link status 1 down cnt 0
2025-04-24T17:56:22.199Z In(182) vmkernel: cpu46:2098180)nfnic: <1>: INFO: fnic_handle_link: 1003: old status 0 old down cnt 0
2025-04-24T17:56:22.199Z In(182) vmkernel: cpu46:2098180)nfnic: <1>: INFO: fnic_handle_link: 1068: fnic1: link up
2025-04-24T17:56:22.199Z In(182) vmkernel: cpu46:2098180)nfnic: <1>: INFO: fnic_fdls_link_status_change: 98: fnic1: FDLS link status change link up:1, usefip:0

FLOGI is sent:

2025-04-24T17:56:22.201Z In(182) vmkernel: cpu46:2098180)nfnic: <1>: INFO: fdls_send_fabric_flogi: 982: Sending fabric FLOGI for wwpn:0x20############88 Setting FLOGI MFS to 2048
2025-04-24T17:56:22.201Z In(182) vmkernel: cpu46:2098180)nfnic: <1>: INFO: fnic_fdls_link_status_change: 113: speed: link_speed: 50
2025-04-24T17:56:22.922Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fdls_process_flogi_rsp: 464: MFS: rdf_size: 2112 iport->mfs: 2048
2025-04-24T17:56:22.922Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fdls_process_flogi_rsp: 473: From fabric: R_A_TOV: 10000 E_D_TOV: 2000
2025-04-24T17:56:22.922Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fnic_fdls_learn_fcoe_macs: 684: learn fcoe: dst_mac: 00:25:b5:##:##:##
2025-04-24T17:56:22.922Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fnic_fdls_learn_fcoe_macs: 689: learn fcoe: fc_mac: 0e:fc:00:##:##:##
2025-04-24T17:56:22.922Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fnic_fdls_set_fcoe_srcmac: 69: Setting src mac: 0e:fc:00:##:##:##
2025-04-24T17:56:22.922Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fnic_fdls_set_fcoe_dstmac: 85: Setting dst_mac: 70:a9:83:##:##:##
2025-04-24T17:56:22.922Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fnic_flogi_reg_handler: 383: FLOGI reg issued fcid 0x5350cd gw_mac 70:a9:83:##:##:##
2025-04-24T17:56:22.923Z In(182) vmkernel: cpu58:2097957)nfnic: <1>: INFO: fnic_fcpio_cmpl_handler: 2765: received FLOGI reg cmpl handler
2025-04-24T17:56:22.923Z In(182) vmkernel: cpu58:2097957)nfnic: <1>: INFO: fnic_fcpio_flogi_reg_cmpl_handler: 1203: FLOGI reg succeeded
2025-04-24T17:56:22.923Z In(182) vmkernel: cpu58:2097957)nfnic: <1>: INFO: fnic_fcpio_flogi_reg_cmpl_handler: 1228: FLOGI REG done. Waking up
2025-04-24T17:56:22.923Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fnic_fdls_register_portid: 1998: FLOGI registration success
2025-04-24T17:56:22.923Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fdls_process_flogi_rsp: 489: Adding vNIC device MAC addr: 0e:fc:00:##:##:##

Next, the PLOGI is attempted:

2025-04-24T17:56:22.923Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:56:22.923Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fdls_send_fdmi_plogi: 1022: FDLS send FDMI PLOGI 0x43##########

2 seconds after sending the PLOGI request, there is a LOGOUT request from the device attach to NPort 0xfffc53:

2025-04-24T17:56:24.506Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fdls_process_logo_req: 3645: Process LOGO request from fcid: 0xfffc53
2025-04-24T17:56:24.506Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fdls_process_logo_req: 3649: Received LOGO req from 0xfffc53 in iport state:3 .Dropping the frame.

The driver then sends fabric ABORTS:

2025-04-24T17:56:26.925Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5

FDMI is retried:

2025-04-24T17:56:26.925Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_fdmi_timer_callback: 2945: retry fdmi timer 1
2025-04-24T17:56:26.925Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fdmi_plogi: 1022: FDLS send FDMI PLOGI 0x43##########
2025-04-24T17:56:30.927Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_fdmi_timer_callback: 2945: retry fdmi timer 2
2025-04-24T17:56:30.927Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fdmi_plogi: 1022: FDLS send FDMI PLOGI 0x43##########
2025-04-24T17:56:33.888Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fnic_fdls_validate_and_get_frame_type: 4121: Receiving Abort Request from s_id: 0xfffc53
2025-04-24T17:56:35.101Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_fdmi_timer_callback: 2945: retry fdmi timer 3
2025-04-24T17:56:35.101Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fdmi_plogi: 1022: FDLS send FDMI PLOGI 0x43##########

Another LOGOUT request from 0xfffc53:

2025-04-24T17:56:35.688Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fdls_process_logo_req: 3645: Process LOGO request from fcid: 0xfffc53
2025-04-24T17:56:35.688Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fdls_process_logo_req: 3649: Received LOGO req from 0xfffc53 in iport state:3 .Dropping the frame.
2025-04-24T17:56:44.778Z In(182) vmkernel: cpu65:2098181)nfnic: <1>: INFO: fnic_fdls_validate_and_get_frame_type: 4121: Receiving Abort Request from s_id: 0xfffc53

PLOGI is issued many more times but these all fail and the FDLS discovery process does not complete so the HBA driver gives up, which leaves the HBA offline:

2025-04-24T17:56:47.099Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:56:51.101Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:57:11.102Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:57:15.104Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:57:35.105Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:57:39.107Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:57:59.109Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:58:03.110Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:58:23.112Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:58:27.114Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:58:47.115Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:58:51.117Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:59:11.118Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_plogi: 1007: Sending fabric PLOGI for wwpn:0x20############88 Setting PLOGI MFS to 2048
2025-04-24T17:59:15.120Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_send_fabric_abts: 955: FDLS sending fabric abts. iport->fabric.state: 5
2025-04-24T17:59:35.122Z In(182) vmkernel: cpu14:2098178)nfnic: <1>: INFO: fdls_error_fabric_disc: 2687: FDLS discovery error from 5 state