One or more HBAs do not come online after upgrade to ESXi 8.0 U1 or 8.0 U2

Products

• VMware vSphere ESX 8.x

Issue/Introduction

One or more HBAs are not discovered following upgrade to ESXi 8.0 U1 (inbox lpfc driver 14.2.560.8-23vmw) or 8.0 U2 (inbox lpfc driver 14.2.641.5-32vmw).

Cause

lpfc driver versions 14.2.560.8 and 14.2.641.5 can support as many as 256 hardware queues (previously 64 were supported). The number of queues created is dependent on the number of CPUs on the ESXi host. 

 

On ESXi servers with larger numbers of CPUs the lpfc heap can be exhausted before queues are created for all HBAs.

 

Sample logging in /var/log/vmkernel.log:

WARNING: Heap: 3646: Heap lpfc already at its maximum size. Cannot expand.
WARNING: Heap: 4106: Heap_Align(lpfc, 8384/8384 bytes, 8 align) failed. caller 0:0x4200411fe97
WARNING: lpfc: _lpfc_HeapAlloc:1612L 3:3309 Alloc failure. Caller <lpfc_sli4_queue_alloc:14990>
WARNING: lpfc: lpfc_alloc_nvme_wq:11070: 3:0509 Failed allocate fast-path NVME WQ (202)
WARNING: lpfc: lpfc_sli4_hba_setup:8120: 3:3089 Failed to allocate queues
WARNING: lpfc: lpfc_pci_probe_one_s4:15474: 3:1421 Failed to set up hba

 

 

 

Resolution

This issue is fixed in ESXi 8.0U3.

Additional Information

To workaround the issue, set the maximum number of nvme I/O queues to 64:

$ esxcli system module parameters set -p "lpfc_nvme_io_channel=64" -m lpfc -a 

A host reboot is required to make the change effective.

 

Note:

On upgrade of ESXi to a build with the fix, the module parameter change will need to be manually reversed:

$ excli system module parameters clear -m lpfc

A host reboot is required to make the change effective.