Hosts in a cluster constantly PSOD every 30- 60 minutes with "Panic Message: @BlueScreen: VERIFY bora/vmkernel/hardware/pci/config.c:157. "

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

PSOD on host with below message

Panic Message: @BlueScreen: VERIFY bora/vmkernel/hardware/pci/config.c:157

Backtrace (top 8 lines):

0x453a5be9b7a0:[0x4200240fa5b0]PanicvPanicInt@vmkernel#nover+0x2cc stack: 0x4200240fa5b0, 0x0, 0x453a5be9b7a0, 0x4200240faa9d, 0x4200240faa9d

0x453a5be9b850:[0x4200240faa9c]Panic_NoSave@vmkernel#nover+0x4d stack: 0x453a5be9b8b0, 0x453a5be9b870, 0x453a5be9ba1c, 0x4200245d6ae0, 0x9d

0x453a5be9b8b0:[0x4200240fb02d]Panic_OnAssertAt@vmkernel#nover+0xba stack: 0x9d00000000, 0x4200245d6ae0, 0x42002463701b, 0x420024647201, 0x420024652340

0x453a5be9b930:[0x42002415071e]Int6_UD2Assert@vmkernel#nover+0x27f stack: 0x0, 0x0, 0x0, 0x420024149068, 0x0

0x453a5be9b960:[0x420024149067]gate_entry@vmkernel#nover+0x68 stack: 0x0, 0x0, 0x0, 0x1, 0xc3701000

0x453a5be9ba20:[0x42002434ce7f]PCIConfigECAMMapDevice@vmkernel#nover+0x7b stack: 0x4c, 0x42002434cf19, 0xffffffff, 0x4c, 0x2

0x453a5be9ba40:[0x42002434cf18]PCIConfig_ECAMRead@vmkernel#nover+0x89 stack: 0x2, 0x0, 0x453a5be9baec, 0x0, 0x1

0x453a5be9ba80:[0x42002435f419]PCIConfig_ReadSafe@vmkernel#nover+0x66 stack: 0x0, 0x372407392f, 0x0, 0x430c1da24ef0, 0x1

Environment

VMware vSphere ESXi 7.0.2

Cause

Many DPDK VM's requesting for memory regions leading to exhaustion of XMAP space leading to PSOD.

Resolution

The issue is resolved in ESXi 7.0.3 release.

Workaround:
There are multiple workarounds:

• Reduce the tx/rx ring sizes of the vmxnet3 vNics.
• Reduce number of VMXNET3 vNics using DPDK VMs which are requesting memory regions.
• Change the driver to e1000.