Module 'DevicePowerOn' power on failed to start the virtual machine after adding/enabling GPU as PCIpassthru device

Products

VMware vSphere ESXi

Issue/Introduction

When adding a GPU to a vSphere VM using PCI passthrough there are a couple of additional settings that need to be done or the VM will not boot.

Check the VM Actions > Edit > VM Options > Boot Options > Firmware and select “EFI”. If this isn't done, the GPUs won’t work and the VM will not boot.
Also, validate the host BIOS settings.
Check and validate the VMDirectPath I/O direct PCI device connections configurations for virtual machines Configuring VMDirectPath I/O pass-through devices on a VMware ESX or VMware ESXi host

Error message:

State
Failed - Module 'DevicePowerOn' power on failed.

Errors

Module 'DevicePowerOn' power on failed.
Failed to start the virtual machine.

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Cause

Error snippets below can be seen in vmware.log:

YYYY-MM-DDTHH:MM:SS In(05) vmx - PCIPassthru: Device ####:##:##.# barIndex 0 type 2 realaddr #x###### size ###### flags 0
YYYY-MM-DDTHH:MM:SS In(05) vmx - PCIPassthru: Device ####:##:##.# barIndex 1 type 3 realaddr #x###### size ###### flags 12
YYYY-MM-DDTHH:MM:SS In(05) vmx - PCIPassthru: Device ####:##:##.# barIndex 3 type 3 realaddr #x###### size ###### flags 12
YYYY-MM-DDTHH:MM:SS In(05) vmx - PCIPassthru: Device has PCI Express Cap Version 2(size 60)
YYYY-MM-DDTHH:MM:SS In(05) vmx - PCIPassthru: Registered a PCI device for ####:##:##.# vIRQ 0x11, physical MSI = Enabled (vmmInt = Enabled),
IntrPin = 1
YYYY-MM-DDTHH:MM:SS In(05) vmx - PCIPassthru: total number of pages needed (######) exceeds limit (######), failing
YYYY-MM-DDTHH:MM:SS In(05) vmx - Module 'DevicePowerOn' power on failed
YYYY-MM-DDTHH:MM:SS In(05) vmx - VMX_PowerOn: ModuleTable_PowerOn = 0
YYYY-MM-DDTHH:MM:SS In(05) svga - SVGA thread is exiting the main loop
YYYY-MM-DDTHH:MM:SS In(05) vmx - Destroying virtual dev for scsi0:0 vscsi=#######

This issue usually happens if there is not enough allocated MMIO space to the VM for all the GPUs.

Resolution

To resolve the issue, it is required to set/configure MMIO (Memory-mapped I/O) parameters for the VM.
Right-click the VM > edit settings, choose VM Options > Advanced > Edit configuration.
Once in the Configuration parameters screen, add two more parameters:

pciPassthru.use64bitMMIO = TRUE
pciPassthru.64bitMMIOSizeGB = "MMIO_Space_In_GB"

Example:
pciPassthru.use64bitMMIO = TRUE
pciPassthru.64bitMMIOSizeGB = 128
Hit OK
Then reload the vmx into inventory using Reloading a vmx file without removing the virtual machine from inventory

Below are the steps to determine the value of "MMIO_Space_In_GB" for the parameter "pciPassthru.64bitMMIOSizeGB":

Set up memory-mapped I/O (MMIO) to map system memory to the GPU’s framebuffer memory so that the CPU can pass data to the GPU.
The 64bitMMIOSizeGB value is calculated by adding up the total GB of frame-buffer memory on all GPUs attached to the VM. If the total GPU framebuffer memory falls on a power-of-2, setting pciPassthru.64bitMMIOSizeGB to the next power of 2 works.
If the total GPU frame-buffer memory falls between two powers-of-2, round up to the next power of 2, then round up again, to get a working setting.

Powers of 2 are 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024

For example, if there is one NVIDIA A100 card with 80GB (in between 64GB and 128GB), round up to the next power of 2 (128GB), then round up again to the next power of 2 after that (256GB) to get the correct setting.

Here are some other useful configurations:

2 x 16GB NVIDIA V100 = 32GB, 32 is a power of 2, so round up to the next power of 2 which is 64, set pciPassthru.64bitMMIOSizeGB = 64 to boot.
2 x 24GB NVIDIA P40 = 48GB, which is in-between 32 and 64, round up to 64 and again to 128, requires pciPassthru.64bitMMIOSizeGB = 128 to boot.
8 x 16GB NVIDIA V100 = 128GB, 128 is a power of 2, so round up to the next power of 2 which is 256, set pciPassthru.64bitMMIOSizeGB = 256 to boot.
10 x 16GB NVIDIA V100 = 160GB, which is in-between 128 and 256, round up to 256 and again to 512, set pciPassthru.64bitMMIOSizeGB = 512 to boot.

Additional Information

Reference Blog: https://blogs.vmware.com/apps/2018/10/how-to-enable-nvidia-v100-gpu-in-passthrough-mode-on-vsphere-for-machine-learning-and-other-hpc-workloads.html
Incorrect value in parameter "pciPassthru.64bitMMIOSizeGB" may also fail to power on the VM, and the below errors can be seen in vmware.log:

Msg_post: Error

[msg.efi.pciMmioError] The Firmware could not allocate ###### KB of PCI MMIO Increase the size of PCI MMIO and try again