ESXi host fails with a diagnostic screen due to an Intel Virtualization Technology Erratum

Products

VMware vSphere ESXi 5.5 VMware vSphere ESXi 5.0 VMware vSphere ESXi 6.0

Issue/Introduction

This article provides information about the problem's cause and the required BIOS fix/Faulty VT-d unit replacement.

The below backtrace can be observed:

cpu2:32999)0x4390c119b660:[0x4180163128c3]VTDQISync@vmkernel#nover+0xf7 stack: 0x1
cpu2:32999)0x4390c119b6a0:[0x4180163137b2]VTDIRWriteIRTE@vmkernel#nover+0x8e stack: 0x2e
cpu2:32999)0x4390c119b6d0:[0x418016313895]VTDIRSteerVector@vmkernel#nover+0x61 stack: 0x43004d129f10
cpu2:32999)0x4390c119b700:[0x4180162e96c9]IOAPICSteerVector@vmkernel#nover+0x59 stack: 0x1c00
cpu2:32999)0x4390c119b740:[0x418016057514]IntrCookie_SetDestination@vmkernel#nover+0x174 stack: 0x4

vmkernel.log

@BlueScreen: IOMMU unit 1 did not complete processing of a queued invalidation wait descriptor after 8 secs. This is likely caused by a known VT-d hardware erratum with a BIOS work-around. Please update the machine's $

Cause

The purple diagnostic screens are rare but can occur in platforms based on these processors:

Intel Xeon Gold Processor 61xx Series
Intel Xeon Processor 55xx Series
Intel Xeon Processor 56xx Series
Intel Xeon Processor 65xx Series
Intel Xeon Processor 75xx Series
Intel Xeon Processor E5-1400 v2 Product Family
Intel Xeon Processor E5-1600 v2 Product Family
Intel Xeon Processor E5-1600 v3 Product Family
Intel Xeon Processor E5-2400 Product Family
Intel Xeon Processor E5-2400 v2 Product Family
Intel Xeon Processor E5-2600 Product Family
Intel Xeon Processor E5-2600 v2 Product Family
Intel Xeon Processor E5-2600 v3 Product Family
Intel Xeon Processor E5-2600 v4 Product Family
Intel Xeon Processor E5-4600 Product Family
Intel Xeon Processor E5-4600 v2 Product Family
Intel Xeon Processor E5-4600 v3 Product Family
Intel Xeon Processor E5-4600 v4 Product Family
Intel Xeon Processor E7-2800 Product Family
Intel Xeon Processor E7-4800 Product Family
Intel Xeon Processor E7-8800 Product Family
Intel Xeon Processor E7-8800/4800/2800 v2 Product Families
Intel Xeon Processor E7-8800/4800 v3 Product Families
Intel Xeon Processor E7-8800/4800 v4 Product Families

The cause of this problem is an Intel IOMMU (VT-d) erratum that causes the IOMMU to stop processing IOTLB invalidation requests submitted by the ESXi host to the IOMMU. This is a non-recoverable condition that causes the ESXi host to fail.

Most Intel Xeon processor based platforms incorporate a work around for the erratum in the machine's BIOS. For platforms that do not, they will require a BIOS update as described in the Resolution section.

Public information about the erratum (BT98) for the Intel Xeon E5 processor family (Sandybridge EN/EP) can be found at Intel Xeon Processor E5 Family spec update.

Information about this erratum in other processor families is not public. Please contact the machine's manufacturer or Intel for support

Resolution

VMWare recommends contacting the hardware manufacturer for updated BIOS or possible workarounds.

Possible workaround can include replacing CPU socket which owns faulty VTD unit. To help in identifying CPU socket refer to boot/vmkernel logs which gives base address of VT-d unit along with VT-d unit number.
e.g "VT-d unit 0: segment 0000 base 0xdc7fc000 ver 1:0 cap 0x8d2078c106f0466 ecap 0xf020df"

Note: A prior version of this KB article recommended that customers experiencing the problem described above work around it by configuring ESXi to disable the Intel VT-d interrupt remapper (setting boot option iovDisableIR=TRUE and rebooting).

VMware ESXi 5.5 p10, 6.0 p04, 6.0 U3 and 6.5 by default disable the Intel VT-d interrupt remapper for this purpose.

VMware has recently received several reports indicating that disabling the Intel VT-d interrupt remapper is causing ESXi host failure on HPE

Gen8 platforms, see ESXi host fails with intermittent NMI PSOD on HP ProLiant Gen8 servers.

VMware is no longer recommending that the Intel VT-d interrupt remapper be disabled to work around the Intel VT-d erratum described in this article.

VMware is recommending that the fix for the erratum be applied in the BIOS as described in the Intel specification updates for the affected processors.

Additional Information

ESXi host fails with intermittent NMI PSOD on HP ProLiant Gen8 servers