PSOD with re-formatting a valid dedup metadata block

Products

• VMware vSAN

Issue/Introduction

This KB is written to advise the necessary steps to take to work around and prevent this issue going forward.

Symptoms:

Host experienced PSOD with following Backtrace

PSOD Backtrace:
Version Details: VMware ESXi 6.7.0 build-16075168
Panic Details: Crash at 2020-07-16T07:22:01.168Z on CPU 27 running world 2098651 - PLOGHelperQueue. VMK Uptime:10:13:03:51.890
Panic Message: @BlueScreen: Re-formatting a valid dedup metadata block
Backtrace:
0x451aa429be10:[0x418037d0bb65]PanicvPanicInt@vmkernel#nover+0x439 stack: 0x451aa42a32c0, 0x418038eb6b08, 0x451aa429bed8, 0x0, 0x431900000001
0x451aa429beb0:[0x418037d0c0a2]Panic_vPanic@vmkernel#nover+0x23 stack: 0xa553c9b2c33be, 0x418037d219c1, 0x431417473b98, 0x418000000010, 0x451aa429bf30
0x451aa429bed0:[0x418037d219c0]vmk_PanicWithModuleID@vmkernel#nover+0x41 stack: 0x451aa429bf30, 0x451aa429bef0, 0x4319f7b62fe0, 0x418000000000, 0x31abf
0x451aa429bf30:[0x418037ceb06a]HelperQueueFunc@vmkernel#nover+0x157 stack: 0x431415efcb18, 0x431415efcb08, 0x431415efcb40, 0x451aa42a3000, 0x431415efcb18
0x451aa429bfe0:[0x418037f107da]CpuSched_StartWorld@vmkernel#nover+0x77 stack: 0x0, 0x0, 0x0, 0x0, 0x0

Environment

VMware vSAN 6.x
VMware vSAN 7.0.x

Cause

This PSOD was introduced to avoid potential data corruption issues in a scenario where certain metadata blocks are bad or in an inconsistent state.

History:
Prior to the vSAN 6.7 release vSAN would re-initialize the block as a bitmap block by discarding any previous allocation in this block, and thus potentially allowing random corruption in user data at a later stage. This panic was added to prevent data from being corrupted.

Future improvements:
In the vSAN 7.0 U1 release, the behavior was changed to more gracefully handle the scenarios and mark the affected diskgroup as offline. This behavior will be ported back to vSAN 6.7 p05 and later. Please see KB 81936 for details.

Resolution

Please contact Broadcom and your hardware vendor to address this issue.

 

Workaround:

Please open a case with Broadcom support to workaround the issue. 

Please update to vSAN 7.0U1 or later to avoid the PSOD, note that the disk group will be taken offline in place of PSOD, please contact Broadcom support and your hardware vendor to address.

Additional Information

Impact/Risks:
The host will be removed from use with the PSOD. This will crash running VMs on the host, and lead to a potential data unavailable or data loss scenario depending on other environmental factors at the time.