PSOD occurs on NVIDIA gpu based ESXI host with "PCPU X locked up. Failed to ack TLB invalidate"

Products

• VMware vSphere ESXi

Issue/Introduction

PSOD screen reports information similar to following: 

PCPU X locked up. Failed to ack TLB invalidate (at least 1 locked up, PCPU(s): X).
PCPU(s) did not respond to NMI. Possible hardware problem; contact hardware vendor.

The vmkernel.log records that the NVIDIA device became unresponsive and reset before the PSOD.

YYYY-MM-DDTHH:MM:SS.536Z cpu4:2097455)WARNING: PCI: 740: Dev ####:##:##.1 is unresponsive after reset
YYYY-MM-DDTHH:MM:SS.154Z cpu8:2097387)WARNING: PCI: 740: Dev ####:##:##.2 is unresponsive after reset

In the logEFI.log located at /var/run/log you see following similar backtrace:

1mVMware ESXi 8.0.3 [Releasebuild-24859861 x86_64]
PCPU 48 locked up. Failed to ack TLB invalidate (at least 1 locked up, PCPU(s): 48).
Module(s) involved in panic: [vmkernel Version Releasebuild-24859861]
*PCPU76:2097285/tlbflushcount
PCPU  0: VSVSVVUSVVVVVVVVVSVVVVVSSVSVVVSVSVSSSSVUVSVIVSSSVUIVSVIUSVVIVSVV
PCPU 64: VSVVVVVVSVVSSSSVSVSVVVVVVVVSSVSVSVVSVSVSSSVSSVVSVSIVVISSSVSVUSUS
cpu76:)Code start: VMK uptime: 90:09:45:22.635
cpu76:):[]PanicvPanicInt@vmkernel#nover+0x20c stack: 
cpu76:):[]Panic_NoSave@vmkernel#nover+0x4d stack: 
cpu76:):[]TLBGetLockedCPUBacktraces@vmkernel#nover+0x265 stack: 0x0
cpu76:):[]TLBDoInvalidate@vmkernel#nover+0x240 stack: 
cpu76:):[]TLBFlushCountForceFlush@vmkernel#nover+0xc4 stack: 0x0
cpu76:):[]CpuSched_StartWorld@vmkernel#nover+0xbf stack: 0x0
cpu76:):[]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0
cpu76:)base fs=0x0 gs= Kgs=0x0
cpu48:)NMI: 738: NMI IPI: PC , SP  (Src 0x1, CPU48)
cpu76:)Possible hardware problem: 2 PCPU(s) [48,49] did not respond to NMI

• This issue may be triggered during network reconvergence or maintenance events on HPE ProLiant Gen10/Gen11 servers using Broadcom BCM57414 adapters.

Environment

• ESXi 8.0.x

• HPE ProLiant DL380 Gen10+ / Gen11
• NVIDIA GPU devices
• Broadcom BCM57414 25GbE adapters

Cause

This issue is caused by an NVIDIA device becoming unresponsive. 

The Physical CPU (PCPU) becomes stuck while attempting to access the PCI configuration space for the NVIDIA device. Because the device is unresponsive, the PCPU fails to acknowledge the Translation Look-aside Buffer (TLB) invalidation request, leading to a system panic.

This causes a PSOD when the PCPU on the same physical core failed to handle the TLB Invalidate Request.

Resolution

Contact NVIDIA support to investigate the cause of the GPU unresponsiveness. Ensure that all NVIDIA drivers and firmware are updated to the latest available versions.

For general troubleshooting of TLB invalidate errors, see Error: Failed to ack TLB invalidate purple diagnostic screen on VMware ESXi. See Understanding a "Failed to ack TLB invalidate" purple diagnostic screen

To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.

Additional Information