NVIDIA BlueField-2 DPU Failure with PCI Fatal Error

Products

• VMware vSphere ESXi

Issue/Introduction

You observe that the NVIDIA BlueField-2 DPU (vmdpu####) encounters an error and becomes unresponsive on the host. The DPU device may not be detected on the PCI bus, and the following symptoms are present in the logs:

Host Status

• The ESXi host reports an error: vmdpu#### (Error) vmdpu#### has encountered an error.
• Uplinks associated with the DPU (e.g., vmnic####, vmnic####) appear as "device not found".

Log Observations (vmkernel.log)

• You see explicit PCI communication and health errors:
• <NMLX_ERR> nmlx5_cc: ####:####:####.#: Health: PCI error detected 
• <NMLX_INF> Device internal error state is set 
• PCIEDPC: ####: ####:####:####.#: Port experienced DPC, reason Fatal error
• The system firmware may report ACPI events (e.g., #### received ACPI event 0xf) followed by PCIe error recovery attempts that fail to restore the device.

System Event Log (SEL)

• The hardware SEL records a critical event coinciding with the failure:
  
  • Assert + Critical Interrupt Bus Fatal Error 

Environment

VMware vSphere ESXi

Cause

The issue is typically caused by a hardware-level failure or a fatal PCIe bus error that triggers Downstream Port Containment (DPC). When a fatal error occurs, the PCIe port is disabled to prevent error propagation, causing the DPU to drop off the bus. While some software-level cosmetic warnings (like Failed to cleanup registration key) may appear in the logs around the same time, they are unrelated to the physical device failure.

Resolution

To resolve this issue, you must perform hardware-level troubleshooting as the device has entered an unrecoverable state for the software driver.

1. Power Cycle the Host: A warm reboot may not be sufficient to reset the PCIe link state. You must perform a cold boot (power cycle) to attempt to re-initialize the DPU hardware.
2. Verify Firmware Compliance: Ensure the DPU and System BIOS/Firmware are at the certified versions for your specific server deployment (e.g., Lenovo, NVIDIA, and VMware joint solutions).
3. Engage Hardware Vendor: If the issue persists after a power cycle or recurs across multiple nodes, you must contact your hardware vendor (e.g., Lenovo) to investigate the physical health of the BlueField-2 card or the PCIe slot.
4. Data Collection for Vendor: To facilitate a root cause analysis by NVIDIA or the hardware vendor, you may be required to install the Mellanox Firmware Tools (MFT) and collect MST dumps or a sysinfo-snapshot while the device is in a failed state.

For more information on similar bus fatal error messages, see: System Event: Message: Assert + Critical Interrupt Bus Fatal Error.