PSOD due to IOMMU fault with VTDIommuFaultHandler or IOMMUProcessFaults in the stack

Products

• VMware vSphere ESXi

Issue/Introduction

PSOD is seen on servers with passthrough/shared graphics cards with either of the following indicatiors

• Crash Message like "@BlueScreen: LINT1/NMI (motherboard nonmaskable interrupt), undiagnosed" with function VTDIommuFaultHandler appearing in the backtrace
• Crash Message like "@BlueScreen: IOMMU Fault detected for 0000:da:00.0 (vmgfx0/xxxx) ..." with function IOMMUProcessFaults appearing in the backtrace.

Cause

This fault is originating from the device/driver related to IO MMU (Input-Output Memory Management Unit). There multiple reason but not limited to

• A failing I/O device
• Driver/firmware issues related to the hardware
• Invalid memory access by the guest using the hardware

Resolution

It is recommended best practice to keep the driver/firmware combination of the device up-to date. Please contact the hardware vendor/OEM for the further diagnostics and resolution.

Additional Information

Identifying the hardware device part of the failure.

PSOD crash message includes the PCI address of the device where fault originated. using lspci command, one can identify the device

Example:

Crash Message: @BlueScreen: IOMMU Fault detected for 0000:da:00.0 (vmgfx0/xxxx) IOaddr: 0xa1fc6a000 Mask: 0x6 Domain: 0x430af4e0c460.

0000:da:00.0 is the PCI address of the device. Running lspci command from shell/ssh session to the ESXi Server will display the device at this address

Command :

lscpic |grep 0000:da:00.0

Output:

0000:da:00.0 3D controller: Vendor Name Device Name [vmgfx0]