Esx 8.0 Host Crashes with DF Exception 8 PSOD

Products

VMware vSphere ESXi

Issue/Introduction

Esx 8.0.3 host crashes with below PSOD

PSOD Backtrace

#0 0x0000420019c6dae1 in CpuSched_PcpuLastLevelCache (pcpu=<optimized out>) at bora/vmkernel/sched/cpusched_int.h:2220
#1 CpuSchedVcpuMigrateScanPackage (longTerm=0 '\000', migType=CPUSCHED_MIGRATE_WAKEUPUNIFY, now=6889412746429732, checkedPcpusPtr=0x45415019bb78,
gcPtr=0x45415019bbf8, bestGoodnessPtr=0x45415019bb08, packagePcpu=<optimized out>, vcpu=0x45413a01f100) at bora/vmkernel/sched/cpusched_rebalance.c:4670
#2 CpuSched_VcpuMigrateBestPcpu (vcpu=vcpu@entry=0x45413a01f100, numRandPcpus=4, migType=migType@entry=CPUSCHED_MIGRATE_WAKEUPUNIFY, now=6889412746429732)
at bora/vmkernel/sched/cpusched_rebalance.c:4876
#3 0x0000420019c6e389 in CpuSched_VcpuWakeupMigrateUnified (vcpu=vcpu@entry=0x45413a01f100, now=now@entry=6889412746429732)
at bora/vmkernel/sched/cpusched_rebalance.c:5005
#4 0x0000420019edb86a in CpuSchedVcpuWakeupMigrate (now=6889412746429732,

vcpu=0x45413a01f100) at bora/vmkernel/sched/cpusched.c:7655
#5 CpuSchedVcpuMakeReady (vcpu=0x45413a01f100, now=6889412746429732) at bora/vmkernel/sched/cpusched.c:4722
#6 0x0000420019edb99d in CpuSchedWakeupWorld (world=<optimized out>, now=6889412746429732) at bora/vmkernel/sched/cpusched.c:8097
#7 0x0000420019edbde6 in CpuSchedWakeupWorldInt (world=world@entry=0x45413a01f000) at bora/vmkernel/sched/cpusched_int.h:1121
#8 0x0000420019edbe0d in CpuSchedWakeupWorldCheck (world=world@entry=0x45413a01f000, shouldDelay=shouldDelay@entry=0 '\000', preemptible=<optimized out>)
at bora/vmkernel/sched/cpusched.c:11093
#9 0x0000420019edbedd in CpuSched_WakeupWorld (world=world@entry=0x45413a01f000) at bora/vmkernel/private/prda.h:577
#10 0x00004200199a6486 in VMKPoll_WakeupAndRemoveWaitersForEvent (waiters=waiters@entry=0x432458739ef0, events=events@entry=VMKPOLL_INMASK)
at bora/vmkernel/main/vmkpoll.c:254
#11 0x0000420019e0f94d in VMKPoll_WakeupAndRemoveWaiters (waiters=0x432458739ef0) at bora/vmkernel/public/vmkpoll.h:145
#12 UserSemaFD_Signal (fd=<optimized out>) at bora/vmkernel/user/userSemaFD.c:646
#13 0x0000420019dd5310 in User_UWVMK64SyscallHandler () at bora/vmkernel/user/user.c:2231
#14 0x0000420019e9edf6 in Syscall_InstructionHandler () at bora/vmkernel/main/x86/common64.S:324
#15 0x0000000000000103 in ?? () from /opt/nfs/vcf_gs_csp_prd_build/build/storage61/release/bora-25067014/publish/debug/vmk/symbols/bnxtnet
#16 0x0000000000000000 in ?? ()

/var/log/vmkernel.log reports below errors

YYYY-MM-DDTHH:MM:SS:06:27.131Z cpu145:2101959)WARNING: Heartbeat: 961: PCPU 269 didn't have a heartbeat for 6 seconds, timeout is 10, 1 IPIs sent; may be locked up.
YYYY-MM-DDTHH:MM:SS:06:27.131Z cpu145:2101959)Heartbeat: 1014: Sending timer IPI to PCPU 269
YYYY-MM-DDTHH:MM:SS:06:27.131Z cpu269:2114648)NMI: 610: 0xfffffffffc407d68:[0xfffffffffc06809a] (vmm64)
YYYY-MM-DDTHH:MM:SS:06:27.131Z cpu269:2114648)NMI: 610: 0xfffffffffc407de8:[0xfffffffffc068475]
YYYY-MM-DDTHH:MM:SS:06:27.131Z cpu269:2114648)NMI: 610: 0xf4240:[0x11]
YYYY-MM-DDTHH:MM:SS:06:27.131Z cpu269:2114648)ALERT: NMI: 738: NMI IPI: PC 0x42000156f1af, SP 0x45414661baa8 (Src 0x1, CPU269)

YYYY-MM-DDTHH:MM:SS cpu0:2097445)World: 3355: PRDA 0x420040000000 ss 0x0 ds 0x750 es 0x750 fs 0x750 gs 0x750
YYYY-MM-DDTHH:MM:SS cpu0:2097445)World: 3357: TR 0x758 GDT 0x454040000888 (0xffff) IDT 0x420001aa4000 (0xffff)
YYYY-MM-DDTHH:MM:SS cpu0:2097445)World: 3359: CR0 0x8001003d CR3 0x612000 CR4 0x14216c
YYYY-MM-DDTHH:MM:SS cpu0:2097445)Panic: 630: Panic from another CPU (cpu 0, world 2097445): ip=0x42000158eb6e randomOff=0x1400000:

Environment

Esx 8.0.3

Cause

This issue is triggered by a malfunctioning processor, which leads to the threads entering an infinite loop

Resolution

We advise reaching out to the hardware vendors to perform a comprehensive diagnostic test and swap out the defective processor or other failing hardware components.