PSOD: #PF Exception 14 in world #######:HELPER_UPLIN IP

Products

• VMware vSphere ESXi
• VMware Cloud Foundation

Issue/Introduction

• ESXi hosts utilizing the Marvell qfle3 driver may experience an unexpected reboot or Purple Screen of Death (PSOD). This crash typically occurs during a driver recovery operation following hardware-level parity errors.
• The host crashes with a backtrace similar to: qfle3_sp_helper_func@(qfle3) ecore_state_wait_enhanced@(qfle3) qfle3_config_rss@(qfle3)
• The following backtrace is observed:

#PF Exception 14 in world HELPER_UPLIN qfle3_sp_helper_func@(qfle3) ecore_state_wait_enhanced@(qfle3) ecore_config_vlan_mac@(qfle3)

• On the ESXI host validate the /var/run/log/vmkernel.log warnings to validate parity errors

WARNING: qfle3: qfle3_parity_attn: [<REDACTED_HOSTNAMES>] Parity errors detected in blocks WARNING: qfle3: _print_next_block: XSEMI

•  Backtrace similar to:

qfle3_sp_helper_func@(qfle3)
ecore_state_wait_enhanced@(qfle3)
ecore_config_vlan_mac@(qfle3)
qfle3_set_mac_one@(qfle3)
qfle3_apply_queue_mac_filter@(qfle3)
qfle3_queue_apply_filter@(qfle3)
UplinkNetq_ApplyHwRxFilter@vmkernel
NetqueueBalActivatePendingFilters@vmkernel
NetqueueBalRxQueueCommitChanges@vmkernel
UplinkNetqueueBal_BalanceCB@vmkernel
UplinkAsyncProcessCallsHelperCB@vmkernel

• Parity errors in vmkernel.log:

yyyy-mm-ddThh:mm:ssZ cpu1:2098258)WARNING: qfle3: qfle3_parity_attn:16971: [vmnic4] Parity errors detected in blocks
yyyy-mm-ddThh:mm:ssZ cpu1:2098258)WARNING: qfle3: _print_next_block:16602: XSEMI
yyyy-mm-ddThh:mm:ssZ cpu1:2098258)WARNING: qfle3: _print_parity:16596:  [0x00000000]
yyyy-mm-ddThh:mm:ssZ cpu1:2098258)WARNING: qfle3: _print_parity:16596:  [0x00000020]

• Backtrace similar to:

qfle3_sp_helper_func (qfle3)
ecore_state_wait_enhanced (qfle3) 
qfle3_config_rss (qfle3)
qfle3_rq_start (qfle3)
qfle3_cmd_start_q (qfle3)
qfle3_sm_q_cmd (qfle3)
qfle3_rx_queue_start (qfle3)
qfle3_queue_start (qfle3)

Environment

• ESXi: 7.x, 8.x, 8.0 Update 3h
• VCF: 5.2.2
• Marvell qfle3 network driver.

Cause

• Parity error is for VFC block.
• The qfle3 driver tried to reference $rax which has a null value. The null-pointer error is an issue within the driver code.
• A race condition exists in the qfle3 driver recovery mechanism. When a hardware parity error triggers a recovery task, the qfle3 helper thread may attempt to access a memory address that has already been freed or is null, resulting in a page fault (#PF).

Resolution

1. See Identifying and Listing Network Interface Cards (NICs) on VMware ESXi.  This information will be needed in Step 2.
2. Validate driver/firmware is up to date per Broadcom Compatibility Guide.
3. Contact the hardware vendor for additional diagnostics on the physical NIC.