ESXi Host PSOD: qedf Driver Memory Race Condition with Unresponsive MFW

Products

• VMware vSphere ESXi

Issue/Introduction

A VMware vSphere ESXi host may experience an unexpected system crash resulting in a Purple Screen of Death (PSOD). The vmkernel diagnostic logs confirm a transition from a hardware firmware hang to a software-induced kernel panic.

Observed Error Messages:

• WARNING: qed: The MFW is not responsive.
• PANIC bora/vmkernel/main/dlmalloc.c:4944 - Usage error in dlmalloc

Environment

VMware vSphere ESXi 8.0

Cause

The root cause is a "double-free" memory race condition within the QLogic/Marvell qedf (FCoE) network driver, version 2.74.1.0. The crash is triggered when the adapter's Management Firmware (MFW) becomes unresponsive, initiating a recovery routine in the driver that fails to properly manage memory references, leading to the observed dlmalloc usage error.

Resolution

To resolve this issue, perform the following steps:

1. Prerequisites: Before performing any maintenance, take a snapshot or backup of the ESXi host configuration.
2. Engage HPE Support: Submit the diagnostic log snippets showing the MFW unresponsive warning and the dlmalloc panic to HPE Support. Request the specific Service Pack for Synergy (SSP) for ESXi 8.0 Update 3 that addresses the qedf cleanup race condition.
3. Upgrade Driver: Upgrade the qedf driver to version 5.4.86.0 or the version validated in the latest HPE Service Pack.
4. Update Firmware: Update the adapter firmware to the version paired with the new driver.
5. Hardware Health Check: Consult with HPE Support to verify the physical integrity of the mezzanine cards.