"AMD IOMMU warning: completion wait bit is not set after a while" error in the vmkernel.log file of the ESXi host

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• You see IOMMU faults if the IOTLB is not properly invalidated.

• You see interrupt steering issues if the interrupt remapping table is not invalidated properly.

• In the /var/log/vmkernel.log file on the ESXi host, you see entries similar to:

AMD IOMMU warning: completion wait bit is not set after a while.

Environment

VMware ESXi 6.5.x
VMware vSphere ESXi 6.7
VMware ESXi 6.0.x
VMware vSphere ESXi 7.0.2

Cause

This issue occurs because the ESXi host does not give enough time for COMPLETION_WAIT commands to complete.

Note: This is limited to AMD IOMMU only. This article is not relevant to Intel IOMMU.

Resolution

This is a known issue affecting VMware ESXi 6.0.x, 6.5.x, and 7.0.2.  Currently, there is no resolution or workaround for ESXI 6.0 and 6.5. 

This is fixed for ESXi 6.7.

For 7.0.2, there is a software issue which can trigger the issue with low probability if system uses DMA map/unmap heavily. This has been been fixed in 7.0.3 release.