vSAN Host crashes with PSOD
search cancel

vSAN Host crashes with PSOD

book

Article ID: 384861

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • vSAN Host hits a PSOD due to a memory checksum issue.

Validation Step:

  • Backtrace :

Panic Details: Crash at 2024-12-07T05:40:45.602Z on CPU 4 running world 2099267 - LFHelper-0x450103f264c0-1. VMK Uptime:9:02:57:04.756
Panic Message: @BlueScreen: Failed at bora/modules/vmkernel/plog/dedup/dedup_util.c:556 -- VMK_ASSERT(((flags & FS_READ_OP) != 0) || DDPIsValidBlkHdr(blkDesc->ctx, blkDesc->pbn, (DDPBlockHeader *)blk))
Backtrace:
  0x45396ba1bc20:[0x42001fcff0a3]PanicvPanicInt@vmkernel#nover+0x327 stack: 0x45396ba1bd18, 0x0, 0x42001fcff0a3, 0x45396ba1bd00, 0x45396ba1bc20
  0x45396ba1bcf0:[0x42001fcff90a]Panic_vPanic@vmkernel#nover+0x23 stack: 0x45bd9bd92dc0, 0x42001fd165ed, 0x45396ba1bd70, 0x453900000010, 0x45396ba1bd70
  0x45396ba1bd10:[0x42001fd165ec]vmk_PanicWithModuleID@vmkernel#nover+0x41 stack: 0x45396ba1bd70, 0x45396ba1bd30, 0x4311b4e014a8, 0x420021687460, 0x420021681600
  0x45396ba1bd70:[0x420021654d11][email protected]#0.0.0.1+0x60e stack: 0x1, 0x45396ba1be98, 0x453900000008, 0x8, 0x1686cb9fffffffff
  0x45396ba1be70:[0x420021656857][email protected]#0.0.0.1+0xe4 stack: 0xff, 0x42002165680e, 0x1, 0x45bd9bd92dc0, 0x0
  0x45396ba1bed0:[0x42002065b4b1][email protected]#1.0.0-0+0x1be stack: 0x450103f271e0, 0x45bd9bd92dc0, 0x450103f20060, 0x420021656774, 0x42002065b4a3
  0x45396ba1bf50:[0x42002065ba47][email protected]#1.0.0-0+0x140 stack: 0x450103f26984, 0x16ba1f000, 0x450103f272a0, 0x450103f273c0, 0x450103f26940
  0x45396ba1bfe0:[0x42001ffb3ad9]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x42001fcc4d80, 0x0, 0x0, 0x0
  0x45396ba1c000:[0x42001fcc4d7f]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0
Running routine processdump.
Searching modules for routine: processdump .. found 244 modules.
Running module: ####################################################################################################################################################################################################################################################
Completed run routine processdump.

  • vmkernel zdump:

ESC[7mYYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)WARNING: PLOG: DDPIsValidBlkHdr:448: PBN 57174608142541, Checksum mismatch: saved 1925093291 computed 3913633082ESC[0m
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)World: 3072: PRDA 0x420041400000 ss 0x0 ds 0xf50 es 0xf50 fs 0xf50 gs 0xf50
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)World: 3074: TR 0xf58 GDT 0x4538c000a000 (0xf77) IDT 0x42001e950000 (0xfff)
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)World: 3075: CR0 0x8001003d CR3 0x202000 CR4 0x10216c
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)Backtrace for current CPU #5, worldID=2099269, fp=0x45396b99bd00
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)0x45396b99bc20:[0x42001e8ff0a3]PanicvPanicInt@vmkernel#nover+0x327 stack: 0x45396b99bd18, 0x0, 0x42001e8ff0a3, 0x45396b99bd00, 0x45396b99bc20
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)0x45396b99bcf0:[0x42001e8ff90a]Panic_vPanic@vmkernel#nover+0x23 stack: 0x45bd9c0d2080, 0x42001e9165ed, 0x45396b99bd70, 0x453900000010, 0x45396b99bd70
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)0x45396b99bd10:[0x42001e9165ec]vmk_PanicWithModuleID@vmkernel#nover+0x41 stack: 0x45396b99bd70, 0x45396b99bd30, 0x4311d2e014a8, 0x420020287460, 0x420020281600
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)0x45396b99bd70:[0x420020254d11][email protected]#0.0.0.1+0x60e stack: 0x420041405601, 0x45396b99be98, 0x453900000008, 0x8, 0xe9454d3affffffff
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)0x45396b99be70:[0x420020256857][email protected]#0.0.0.1+0xe4 stack: 0xff, 0x42002025680e, 0xf50, 0x45bd9c0d2080, 0x0
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)0x45396b99bed0:[0x42001f25b4b1][email protected]#1.0.0-0+0x1be stack: 0x450103f26e40, 0x45bd9c0d2080, 0x450103f20060, 0x420020256774, 0x42001f25b4a3
YYYY-MM-DDTHH:MM.SSSZZ cpu5:2099269)0x45396b99bf50:[0x42001f25ba47][email protected]#1.0.0-0+0x140 stack: 0x450103f26984, 0x16b99f000, 0x450103f26f00, 0x450103f27020, 0x450103f26940
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)0x45396b99bfe0:[0x42001ebb3ad9]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x42001e8c4d80, 0x0, 0x0, 0x0
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)0x45396b99c000:[0x42001e8c4d7f]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0
YYYY-MM-DDTHH:MM.SSSZ cpu5:2099269)ESC[45mESC[33;1mVMware ESXi 7.0.3 [Releasebuild-21424296 x86_64]ESC[0m
Failed at bora/modules/vmkernel/plog/dedup/dedup_util.c:556 -- VMK_ASSERT(((flags & FS_READ_OP) != 0) || DDPIsValidBlkHdr(blkDesc->ctx, blkDesc->pbn, (DDPBlockHeader *)blk))

Environment

VMware vSAN 6.x
VMware vSAN 7.x
VMware vSAN 8.x

Cause

  • Single-bit alteration in the memory metadata at a specific offset which resulted in a checksum mismatch between the expected and actual values, leading to PSOD on the host.
  • This metadata corruption is typically caused by hardware issues.

Resolution

Engage hardware vendor to identify the root cause and resolve the issue effectively.

Powered by Wolken Software