ESXi Host PSOD due to Race Condition in HBR Replication Filter Teardown

Products

• VMware

Issue/Introduction

This article describes a PSOD event occurring on ESXi hosts utilizing Host Based Replication (HBR). The crash is caused by a race condition during the replication filter teardown process.

• ESXi host encounters a Purple Screen of Death (PSOD).
• Backtrace indicates the panic occurs during replication filter operations.

VMware ESXi 8.0.3 [Releasebuild-24784735 x86_64]
NMI IPI: Panic requested by another PCPU. PC 0x42001c290e24, SP 0x453b8931ba38 (Src 0x4, CPU66)
cr0=0x80050031 cr2=0x20000630034 cr3=0x11433304000 cr4=0x156668
FMS=06/6a/6 uCode=0xd0003f5
*PCPU66:155936150/vmm0:###
PCPU  0: SUVVVVVVVVVVVVVSVVVVVVVVVUVVVVVVVVVVVVVSVSVSSVVVSUSVVSVVVVSVVVS
PCPU 64: VSVVSUVVSSVVSUVS
Code start: 0x42001c200000 VMK uptime: 226:05:05:41.488
Saved backtrace from: pcpu 66 SpinLock spin out NMI
0x453b8931ba38:[0x42001c290e23]Power_ArchPerformWait@vmkernel#nover+0xd4 stack: 0x420050801880
0x453b8931ba40:[0x42001c290f75]Power_ArchSetCState@vmkernel#nover+0xba stack: 0x100000020
0x453b8931ba90:[0x42001c8da1ed]CpuSchedIdleLoopInt@vmkernel#nover+0x292 stack: 0x42
0x453b8931bb00:[0x42001c8de718]CpuSchedDispatch@vmkernel#nover+0x1e31 stack: 0x452200000001
0x453b8931bd40:[0x42001c8df12a]CpuSchedWait@vmkernel#nover+0x35b stack: 0x8000000000000007
0x453b8931beb0:[0x42001c8df748]CpuSchedVcpuHaltWork@vmkernel#nover+0x195 stack: 0x1
0x453b8931bf00:[0x42001c8b369e]VMMVMKCall_Call@vmkernel#nover+0x103 stack: 0x0
0x453b8931bfd0:[0x42001c8b00d0]VMKVMM_ArchEnterVMKernel@vmkernel#nover+0x21 stack: 0x0
base fs=0x0 gs=0x420050800000 Kgs=0x0
1 other PCPU is in panic.

Environment

VMware ESXi 8.x

Cause

When replication filters detach and re-attach in tight bursts, the retry timer may fire precisely when the object is being torn down, leading to a system crash.

Resolution

To immediately restore stability without downtime, apply the following workaround:

1. Reboot the ESXi host.
2. SSH to the host.
3. Run the following command to increase the retry delay: esxcli system settings advanced set -o /HBR/ReconnectFailureDelaySecs -i 120
4. Run the following command to cap the maximum delay: esxcli system settings advanced set -o /HBR/ReconnectMaxDelaySecs -i 300