LPFC driver is not detecting all HBAs on ESXi

Products

VMware vSphere ESXi 8.0

Issue/Introduction

ESXi server has three or more Emulax HBAs

ESXi is running or was upgraded to 8.0U1 or 8.0U2

From vmkernel log, we can see lpfc driver fails to initialize the device because of out-of-memory - "Heap lpfc already at its maximum size".

2023-09-14T16:36:35.427Z cpu179:2099325)lpfc: lpfc_map_topology:10582: 3:2020 Using persistent topology value [P2P Only]
2023-09-14T16:36:35.427Z cpu179:2099325)lpfc: lpfc_sli4_read_config:10755: 3:2003 cfg params Extents? 0 XRI(B:6145 M:6144), VPI(B:273 M:272) VFI(B:21 M:20) RPI(B:6145 M:6144) FCFI(Count:17) conf_trunk:x0
2023-09-14T16:36:35.447Z cpu179:2099325)lpfc: lpfc_sli4_hba_setup:7981: 3:4003 Dual Dump is enabled
2023-09-14T16:36:35.460Z cpu179:2099325)lpfc: lpfc_sli4_xri_scsi_sgl_update:4161: 3:2401 Current allocated IO xri-sgl count:0, maximum SCSI xri count:5894
2023-09-14T16:36:35.468Z cpu179:2099325)WARNING: Heap: 3646: Heap lpfc already at its maximum size. Cannot expand.
2023-09-14T16:36:35.468Z cpu179:2099325)WARNING: Heap: 4106: Heap_Align(lpfc, 8384/8384 bytes, 8 align) failed. caller: 0x42000411fe97
2023-09-14T16:36:35.468Z cpu179:2099325)WARNING: lpfc: _lpfc_HeapAlloc:1612: 3:3309 Alloc failure. Caller <lpfc_sli4_queue_alloc:14990>
2023-09-14T16:36:35.468Z cpu179:2099325)WARNING: lpfc: lpfc_alloc_nvme_wq:11070: 3:0509 Failed allocate fast-path NVME WQ (202)
2023-09-14T16:36:35.468Z cpu179:2099325)WARNING: lpfc: lpfc_sli4_hba_setup:8120: 3:3089 Failed to allocate queues
2023-09-14T16:36:35.468Z cpu179:2099325)WARNING: lpfc: lpfc_pci_probe_one_s4:15474: 3:1421 Failed to set up hba
2023-09-14T16:36:35.541Z cpu58:2099328)lpfc: lpfc_scsidone_event:6835: 3:3285 ScsiDone thread death pending....
2023-09-14T16:36:35.541Z cpu58:2099328)lpfc: lpfc_scsidone_event:6844: 3:3286 ScsiDone thread has stopped
2023-09-14T16:36:35.542Z cpu98:2099327)lpfc: lpfc_pathclaim_event:12776: 3:(0):3258 PathClaim thread death pending....
2023-09-14T16:36:35.542Z cpu98:2099327)lpfc: lpfc_pathclaim_event:12784: 3:(0):3260 PathClaim thread has stopped
2023-09-14T16:36:35.542Z cpu179:2099325)lpfc: destroy_port:5421: 3:(0):9035 Destroying Physical port 0x431796ebf0e0 refcnt 1
2023-09-14T16:36:35.542Z cpu179:2099325)DMA: 766: DMA Engine 'lpfc_adapter_dma_engine_3' destroyed.
2023-09-14T16:36:35.542Z cpu24:2099326)lpfc: lpfc_do_work_event:12900: 3:3228 lpfc_do_work_event stopped : state 4
2023-09-14T16:36:35.542Z cpu179:2099325)DMA: 766: DMA Engine 'drb_dma_eng' destroyed.
2023-09-14T16:36:35.542Z cpu179:2099325)DMA: 766: DMA Engine 'hrb_dma_eng' destroyed.
2023-09-14T16:36:35.542Z cpu179:2099325)DMA: 766: DMA Engine 'lpfc_descriptor_eng' destroyed.
2023-09-14T16:36:35.542Z cpu179:2099325)DMA: 766: DMA Engine 'coherent_dma_eng' destroyed.
2023-09-14T16:36:35.542Z cpu179:2099325)DMA: 766: DMA Engine 'lpfc_mbuf_dma_eng' destroyed.
2023-09-14T16:36:35.542Z cpu179:2099325)DMA: 766: DMA Engine 'lpfc_scsi_dma_eng' destroyed.
2023-09-14T16:36:35.542Z cpu179:2099325)WARNING: lpfc: lpfc_attachDevice:15844: failed pci_probe_one: Failure dev 0x2c51430ed10111fb
2023-09-14T16:36:35.542Z cpu179:2099325)Device: 2753: Module 47 did not claim device 0x2c51430ed10111fb.

Environment

VMware vSphere ESXi 8.0U1

VMware vSphere ESXi 8.0U2

Cause

See KB Broadcom Emulex HBA port(s) not initialized when there are multiple HBAs on an ESXi server

Resolution

Upgrade to 8.0U3

If upgrade is not possible at this time then run command esxcli system module parameters set -p "lpfc_nvme_io_channel=64" -m lpfc on all hosts in the cluster to set lpfc module parameter 'lpfc_nvme_io_channel' to 64 and then reboot the hosts.

Once upgraded to 8.0U3 clear the lpfc module parameter by running esxcli system module parameters clear -m lpfc and reboot the hosts.

Additional Information

esxcli system Commands