Host Crashes with PSOD with VM's having multiple PCIPassthru devices enabled

Products

• VMware vSphere ESXi

Issue/Introduction

• Host crashes with below PSOD with VM's having multiple PCIPassthru devices enabled

Module(s) involved in panic: [vmkernel Version Releasebuild-24755229]
cr0=0x80050033 cr2=0x38 cr3=0x81536db000 cr4=0x152660
FMS=06/ad/1 uCode=0x1000404
frame=0x4539e75dbc60 ip=0x42002d2a01aa err=0x0 rf lags=0x10002
rax=0x1 rbx=0x42005280a380 rcx=0x400
rdx=0x4b rbp=0x1 rsi=0x0
rdi=0x4539eb95f000 r8=0x452184780000 r9=0x3
r10=0x42002d2a0154 r11=0x0 r12=0x42005280a390
r13=0x4313abc01430 r14=0x0 r15=0x0
*PCPU74:2298688/vmm15:rhe19.7 (VMM)
PCPU 0: UUVIVSVUVSV IVSVUVSVUVUVUVSVSVSVSUVVSVSVS ISVIVIVIVSVUVIVSVIVIVIVI
PCPU 64: VIVISIVIVSVIVIVIVIVSVIVIVIVISVVSVIVIVIVIVIVIVIVIVIVIVIVIVIISIIIU
IIIIIIIIIIIIIIUSSSSUUISUUUUSSUUUUI
PCPU192: SISIUSUUUISUUUUUISSUIUUUUIIUIIUUUIUIIUII
IUIIIUIIIUIIIIIISIII

Code start: 0x42002d200000 VMK uptime: 6:08:47:02.900
0x4539e75dbd20: [0x42002d2a01aa ]VAPICIntrVirtHandler@vmkernel#nover+0x56 stack: 0x0
0x4539e75dbd50 : [0x42002d376069]IntrCookie_Do Interrupt@vmkernel#nover+0x1a2 stack: 0x0
0x4539e75dbe00: [0x42002d376578]IntrCookie_Vmkernel Interrupt@vmkernel#nover+0x3d stack: 0xf3
0x4539e75dbe20: [0x42002d8dd395]IDT_IntrHandler@vmkernel#nover+0x9a stack: 0x0
0x4539e75dbe40 : [0x42002d8d60ce]gate_entry@vmkernel#nover+Oxaf stack: 0x0
0x4539e75dbf00:[0x42002d8e91eb]VMMVMKCall_Call@vmkernel#nover+0x277 stack: 0x0
0x4539e75dbfd0: [0x42002d8e579d]VMKVMM_ArchEnterVMKerne1@vmkernel#nover+0x26 stack: 0x0

• This crash occurs when  hot remove operation is performed, device unmap MMIO fails and VM panic triggers. 
• During this VM shutdown, all the passthru devices are being unregistered, while the devices un-registration is happening, if another device is hot removed.

 

Environment

Esx 9.0

Cause

This issue occurs due to a race condition while the VM is shutting down, all the passthru devices device are being un-registered.

Resolution

Broadcom Engineering is aware of this and would be releasing a fix in the upcoming release.

Workaround: 

• Add this parameter in the vmx advanced settings: pciPassthru.allowP2P = false
• Ref:Configure Virtual Machine Advanced File Parameters

Additional Information

Configuring Virtual Machine Advanced Parameter property using OVF template