Esx Host crashes with PSOD

Products

• VMware vSphere ESXi

Issue/Introduction

• Esxi host crashes with PSOD
• /var/log/vmkernel.log reports below errors 
  YYYY-MM-DDTHH:MM:SS.389Z cpu35:3937182)MCA: 202: CE Intr G0 Be S88000040020000b1 A0 M900000000000000 P0/0 Memory Controller Address/Command Error on Channel 1.
YYYY-MM-DDTHH:MM:SS.389Z cpu36:3294049)MCA: 202: CE Intr G0 Be Sc8000180020000b1 A0 M900000000000000 P0/0 Memory Controller Address/Command Error on Channel 1.

YYYY-MM-DDTHH:MM:SS.389Z cpu34:4635225)MCA: 202: CE Intr G0 B7 S8c00004001010091 A448c9caa80 M406c30f800886 P448c9caa80/40 Memory Controller Read Error on Channel 1.
} =
MCA error class: CE = Corrected
Reported by: Intr = Corrected Machine Check Interrupt (CMCI) handler.
MCG_STATUS: 0x0000000000000000
MCA bank number: 0x7
MCi_STATUS: 0x8c00004001010091
   MCACOD (architectural) 0x0091 = Memory Controller Read Error on Channel 1.

• We may expect below backtrace

Sched_SysServiceStartSpecial (type=type@entry=246, data=data@entry=0) at bora/vmkernel/sched/sched_sysacct.c:1357
0x000042001649ff67 in Sched_SysServiceStart (type=type@entry=246) at bora/vmkernel/sched/sched_sysacct.c:1438
0x000042001616174f in IntrCookieCallHandler (cookie=<optimized out>, clientData=0x0, handler=<optimized out>) at bora/vmkernel/main/intrCookie.c:2018
IntrCookie_DoInterrupt (vector=246, vectorData=vectorData@entry=0, fullFrame=fullFrame@entry=0x453b5ae1be20) at bora/vmkernel/main/intrCookie.c:4170
0x00004200166a9b25 in IDT_ForwardIntr (vector=vector@entry=246) at bora/vmkernel/main/x86/idt.c:1929
0x00004200166a9b5b in IDT_VMMForwardIntr (function=<optimized out>, args=<optimized out>) at bora/vmkernel/main/x86/idt.c:1941
0x00004200166b469f in VMMVMKCall_Call (function=33, args=0xfffffffffc407ec0, eflagsBefore=eflagsBefore@entry=150, status=status@entry=0x453b5ae1bfdc) at bora/vmkernel/main/x86/vmmvmkcall.c:712
0x00004200166b10d1 in VMKVMM_ArchEnterVMKernel (fnNum=<optimized out>, args=<optimized out>, eflags=150) at bora/vmkernel/main/x86/vmkvmm.c:336
0xfffffffffc06809a in ?? ()
0x0000000000000000 in ?? ()

Environment

Esxi 8.0.3

Cause

Analysis of the memory dump indicates that the PSOD was triggered when the processor retrieved an unexpected value of 0x0 from schedSysServiceStats[25]->accounting[246]. Curiously, the valid value was still present in memory.

Resolution

To fix this issue it is recommended to contact the hardware vendors(OEM) to run an intense hardware diagnosis and replace the faulty memory components.