ESXi encountered a PSOD after upgrading qfle3 to version 1.4.46.0

Products

• VMware vSphere ESXi

Issue/Introduction

• After upgrading the qfle3 driver to version 1.4.46.0 using the command below
  
  esxcli software vib install -v /tmp/QLC_bootbank_qfle3_1.4.46.0-1OEM.700.1.0.15843807.vib
  
  ESXi encounters a Purple Screen of Death (PSOD)
  
  #PF Exception 14 in world 2097970:vmkwdevmgr IP #### addr 0x####
PTEs:0x100106003:0x21d1561063:0x0:
cr0=0x8001003d cr2=0x45211642000 cr3=0x103e94000 cr4=0x406e70
frame=0x45ce8b8950b ip=0x#### err=0x00000004 rflags=0x10246
rax=0x0 rbx=0x45ce8b8962d rcx=0x45ce8b89620 rdx=0x0
rsi=0x0 rdi=0x45ce8b895f0 rbp=0x45ce8b89600 r8=0x7
r9=0x0 r10=0x0 r11=0x0 r12=0x45ce8b89600
r13=0x45ce8b8960c r14=0x45ce8b89638 r15=0x45ce8b89618
*PCPU0:2097970/0x4 vmm0: IDLE
PCPU1:USSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Code start:0x420006500000 VMK uptime: 0:00:03:03.090


• Following the PSOD, when ESXi is rebooted, the qfle3 driver automatically reverts to its original version..
  
  
• From the output of command localcli software vib list, it shows qcnic, qfle3, qfle3i, and qfle3f are provided by different vendors.
  
qcnic                          ####          QLC      VMwareCertified
qfle3                          ####          QLC      VMwareCertified
qfle3i                         ####          QLC      VMwareCertified
qfle3f                         ####          VMW      VMwareCertified


• In /var/run/log/esxupdate.log, the below errors are observed.
  
  esxupdate: ge/ImageManager/HostSeeding.py", line 1076, in cacheVibs     self.addVib(vib, hostImage, deployDir)   File "/lib64/python3.8/site-packages/vmware/esximage/ImageManager/HostSeeding.py", line 1134, in addVib     vibDownloadFn(cachePath, vib, checkdigests=True,   File "/lib64/python3.8/site-packages/vmware/esximage/Depot.py", line 938, in GenerateVib     raise Errors.VibRecreateError(vibobj.id, msg) vmware.esximage.Errors.VibRecreateError: ('VMW_bootbank_qfle3i_####-15vmw.####', 'Cannot find or stage payload qfle3i type vgz of VIB VMW_bootbank_qfle3i_####-15vmw.####: VMW_bootbank_qfle3i_####-15vmw.####: Cannot find VIB VMW_bootbank_qfle3i_####-15vmw.#### in BootbankInstaller')
  

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

The issue occurs because the qfle3 driver installation is not persistent after a PSOD and reboot. The mismatch of driver vendor sources (qcnic, qfle3, qfle3i, qfle3f) may lead to inconsistencies and failures during driver upgrade or validation.

Resolution

1. If the ESXi boot bank relies on a local disk connected via an HBA card, confirm that the HBA card driver is not part of the qfle3 series.
   If the boot disk HBA card uses a qfle3-series driver, stop and contact Broadcom for further troubleshooting.
   Use the following command to check the driver of the HBA card:

   [root@localhost:~] esxcli storage core adapter list

   Output:

   HBA Name  Driver    Link State  UID          Capabilities  Description
--------  --------  ----------  -----------  ------------  -----------
vmhba0    ####        ####      ####              ####       ####

2. Place the ESXi host into maintenance mode.
   
   
3. Remove the qcnic, qfle3, qfle3i, and qfle3f VIBs using the following command:
   
   esxcli software vib remove -n <VIB_Name>
   
   
4. Install the qcnic, qfle3, qfle3i, and qfle3f VIBs using the following command. Ensure all qfle3-related drivers (qcnic, qfle3, qfle3i, qfle3f) are installed from the same vendor source to avoid conflicts.
   
   esxcli software vib install -n <VIB_Name>