System fails to power on a VM with VMDirect Path I/O device, reporting interrupt allocation failures

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• System fails to power on a VM when multiple VMDirect Path I/O devices are configured.
• The vmkernel logs contain entries similar to:

vmkernel: cpu124:2102458)PCIPassthru: 2513: failed to allocate INTx interrupt
OR
VMKPCIPassthru: VMKPassthruAllocateIntrCookies:1769: failed to allocate MSI interrupt

Environment

VMware vSphere ESXi 7.0.3
VMware vSphere ESXi 7.0.2

Cause

System has a limited global interrupt (INTx and MSI) vector pool. Global pool is shared across different devices on the platform including passthrough devices. When there are many devices requesting global interrupts than what the system has in its global pool, allocation failures are reported.

Resolution

To resolve this issue, increase the value of boot option "intrFirstMsixVector" from its default 0x70 to a larger value - 0x80 or 0x90.

Note:

• The total number of interrupt vectors on a platform is fixed and hence increasing the size of global vectors will decrease the number of available MSIx vectors.
• This resolution is applicable from ESXi 7.0.1 until ESXi 8.0. For 8.0U1 and beyond, this is not needed due to global vector pool optimization.
•  For more information on instructions to configure boot options, please refer Configuring advanced options for ESXi/ESX

Workaround: