ESXi host experiences a PSOD due to corruption of the USB device attached

Products

• VMware vSphere ESXi

Issue/Introduction

• An ESXi host experiences recursive PSOD similar to the below image:
  
  
  
• On the ESXi host, in /var/run/log/LogEFI.log:

YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)VMware ESXi <version and build>
#DE Exception 0 in world 2098####:VCHelperQueu @ 0x42000531####
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)cr0=0x8001003d cr2=0x63950170a0 cr3=0x60f000 cr4=0x10216c
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)FMS=06/3f/2 uCode=0x49
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)frame=0x4538e0d1bc20 ip=0x42000531#### err=0x0 rflags=0x10246
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)rax=0x80 rbx=0xbad000e rcx=0x200
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)rdx=0x0 rbp=0x430ea1841b00 rsi=0x323
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)rdi=0x0 r8=0x80 r9=0x80
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)r10=0x1 r11=0x430ea1800e58 r12=0x4308f4700828
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)r13=0x430ea1841b00 r14=0x430ea1841bb0 r15=0x430ea1841b28
*PCPU21:2098####/VCHelperQueue
PCPU  0: USUISSUSSSSSSSUISUIISSUSSSIISUUIUUUIUIIS
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)Code start: 0x420004800000 VMK uptime: 0:00:00:SS
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)0x4538e0d1bce0:[0x42000531####]VFATAnalyzeDiskHeader@(vfat)#<None>+0x177 stack: 0x4308fb7e8f80
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)0x4538e0d1bd00:[0x4200053128ad]VFAT_Open@(vfat)#<None>+0x25e stack: 0x4308f46f4740
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)0x4538e0d1bd70:[0x42000483b184]FSSVec_OpenFS@vmkernel#nover+0x2d stack: 0x41ffc48acd60
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)0x4538e0d1bd80:[0x4200048347be]FSSDefaultProbe@vmkernel#nover+0x39b stack: 0x4308f46f99f0
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)0x4538e0d1be10:[0x420004834b2e]FSS_Probe@vmkernel#nover+0x4f stack: 0x4308f46fb7f0
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)0x4538e0d1be80:[0x420004849af2]VCProbeDeviceAndUpdateState@vmkernel#nover+0x253 stack: 0x41ffc48ad380
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)0x4538e0d1bef0:[0x420004849b92]VCProbeViaHelper@vmkernel#nover+0x57 stack: 0x4538e1a9b460
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)0x4538e0d1bf40:[0x4200048da519]HelperQueueFunc@vmkernel#nover+0x1d2 stack: 0x4538e0d20b48
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)0x4538e0d1bfe0:[0x420004bb4ebd]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)0x4538e0d1c000:[0x4200048c4fff]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0
YYYY-MM-DDTHH:MM:SSZ cpu21:2098####)base fs=0x0 gs=0x420045400000 Kgs=0x0

• The affected ESXi host has a USB device attached to access KVM.

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

• The PSOD occurs due to corruption in the bootSector read on the USB device when the kernel tries to access the vFAT volume.

Resolution

• Detach the USB device to successfully boot the ESXi host.
• Perform a complete health check of the KVM USB device and replace if required.