Broadcom Emulex cards are not detected after ESXi upgrade to 7.0 U3

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• After upgrading ESXi hosts to 7.0 U3, the Emulex FC cards are not detected.
  
  
• Error: The number of vSphere HA heartbeat datastores for this host is 0, which is less than required: 2

Environment

VMware vSphere ESXi 

Cause

The following HBA models have gone End of Support and are deprecated in the 14.0.x.x drivers (For OCe14000-series VMware 7.0 U3 inbox driver will be 12.0.1500.2)

Refer - https://www.broadcom.com/support/knowledgebase/3211238682009/broadcom-emulex-end-of-life-hba-models-not-detected-with-14.0.x.x-and-later-drivers 

• Emulex LPe12000-series 8Gb/s PCIe 2.0 FC HBA
• Emulex LPe16000-series 16Gb/s PCIe 2.0 FC HBA
• Emulex OCe14000-series 10GbE FCoE Adapter

 

 

Resolution

• Compare the upgraded host with the non-upgraded host.
  
  In below example Host 1 is upgraded to ESXi 8.x and Host 2 is still running on ESXi 7.x . Both the hosts are running the same hardware model.

Run - "esxcli storage core adapter list" on both host to compare the vmhba adapters detected by the host. We can see that lpfc adapter is missing on Host 1. 

Host 1

[root@localhost01:~] esxcli storage core adapter list
HBA Name  Driver     Link State  UID               Capabilities           Description
--------  ---------  ----------  ----------------  -------------------    -----------
vmhba0    lsi_mr3    link-n/a    sas.####                                 (0000:15:00.0) Broadcom MegaRAID SAS Fury Controller
vmhba64   qfle3i     unbound     iscsi.vmhba64     Second Level Lun ID    QLogic 57840 10 Gigabit Ethernet Adapter
vmhba65   qfle3i     link-down   fcoe.####         Second Level Lun ID    () QLogic Inc. FCoE Adapter
vmhba66   qfle3i     unbound     iscsi.vmhba66     Second Level Lun ID    QLogic 57840 10 Gigabit Ethernet Adapter
vmhba67   qfle3i     link-down   fcoe.####         Second Level Lun ID    () QLogic Inc. FCoE Adapter

[root@localhost01:~] vmware -vl 
VMware ESXi 8.0.2 build-23305546
VMware ESXi 8.0 Update 2

 

Host 2

[root@localhost02:~] esxcli storage core adapter list
HBA Name  Driver     Link State  UID               Capabilities           Description
--------  ---------  ----------  ----------------  -------------------    -----------
vmhba0    lsi_mr3    link-n/a    sas.####                                 (0000:15:00.0) Broadcom MegaRAID SAS Fury Controller
vmhba1    1pfc       link-up     fc.####           Second Level Lun ID    (0000:0b:00.0) Emulex Corporation Emulex LPe12000 8Gb PCIe Fibre
vmhba2    lpfc       link-up     fc.####           Second Level Lun ID    (0000:0b:00.1) Emulex Corporation Emulex LPe12000 8Gb PCIe Fibre
vmhba64   qfle3i     unbound     iscsi.vmhba64     Second Level Lun ID    QLogic 57840 10 Gigabit Ethernet Adapter
vmhba65   qfle3i     link-down   fcoe.####         Second Level Lun ID    () QLogic Inc. FCoE Adapter
vmhba66   qfle3i     unbound     iscsi.vmhba66     Second Level Lun ID    QLogic 57840 10 Gigabit Ethernet Adapter
vmhba67   qfle3i     link-down   fcoe.####         Second Level Lun ID    () QLogic Inc. FCoE Adapter

[root@localhost02:~] vmware -vl 
VMware ESXi 7.0.3 build-24585291
VMware ESXi 7.0 Update 3

 

• Run "lspci | less" on Host 1 and validate that the physical Emulex adapters are getting initialized during boot.

0000:0b:00.0 Fibre Channel: Emulex Corporation LPe12000 Series 8Gb Fibre Channel Adapter
0000:0b:00.1 Fibre Channel: Emulex Corporation LPe12000 Series 8Gb Fibre Channel Adapter

Workaround:

• Install the compatible 12.8.x.x drivers for version 7.0 U3 according to VMware HCL - Requires ESXi reboot after installation.
  
  
  • Run "vmkchdev -l | grep vmhba1" to get the model details, which is in use by Host 2.
  • Refer Determining Network/Storage firmware and driver version in ESXi and validate the model using the Broadcom Support Portal.
  • Follow KB  to download driver version 12.8.X for lpfc adapter.
  • Run "esxcli software vib install –v /path/*.vib" to install the vib after placing the host in maintenance mode.

[root@localhost01:/vmfs/volumes/5804d0a5-########-####-000eleab18e0/Driver/vib20/lpfc] esxcli software vib install -v /vmfs/volumes/5804d0a5-########-####-000eleab18e0/Driver/vib20/lpfc/EMU_bootbank_lpfc 12.8.614.3-10EM. 700.1.0.15843807.vib

Installation Result

Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

VIBs Installed: EMU_bootbank lpfc 12.8.614.3-10EM. 700.1.0.15843807

VIBs Removed: VMW_bootbank_lpfc 14.2.641.5-32vmw. 802.0.0.22380479

VIBs Skipped:

Reboot Required: true

DPU Results:

Note: Now the prompt on the host for HA heartbeat datastore also got fixed since now we have more than the required 2 shared volumes on host 625.
Refer - HA error: "The number of heartbeat datastores for host is 1, which is less than required: 2"

or, 

• Upgrade to 16Gb/s LPe31000-series or 32Gb/s LPe32000-series PCIe 3.0 FC HBA or newer Emulex FC HBA.
  
  
• There is no upgrade path for the OCE14000 10GbE FCoE Adapter.

 

Additional Information

Impact/Risks:

After upgrading to ESXi 7U3, the HBA cards will not be detected under the "Storage Adapters" and the datastores will be missing on the host.