Module 'DevicePowerOn' power on failed to start the virtual machine after adding/enabling GPU as PCIpassthru device

Products

VMware vSphere ESXi

Issue/Introduction

When adding a GPU to a vSphere VM using PCI passthrough there are a couple of additional settings that need to be done or the VM won’t boot.

Check the VM Actions > Edit > VM Options > Boot Options > Firmware and select “EFI”. If this isn't done, the GPUs won’t work and the VM won’t boot.
Also, validate the host BIOS settings
Check and validate the VMDirectPath I/O direct PCI device connections configurations for virtual machines Configuring VMDirectPath I/O pass-through devices on a VMware ESX or VMware ESXi host

Error message:

State
Failed - Module 'DevicePowerOn' power on failed.

Errors

Module 'DevicePowerOn' power on failed.
Failed to start the virtual machine.

Environment

VMware vSphere ESXi 7.0

Cause

The power on of the VM fails with the following error in vmware.log:

PCIPassthru: total number of pages needed (XXXXXXX) exceeds the limit (XXXXXXX), failing

Module 'DevicePowerOn" power on failed

Error snippets below can be seen in vmware.log:

YYYY-MM-DDTHH:MM:SS In(05) vmx - PCIPassthru: Device 0000:5e:00.0 barIndex 0 type 2 realaddr 0xc4000000 size 16777216 flags 0
YYYY-MM-DDTHH:MM:SS In(05) vmx - PCIPassthru: Device 0000:5e:00.0 barIndex 1 type 3 realaddr 0x33000000000 size 34359738368 flags 12
YYYY-MM-DDTHH:MM:SS In(05) vmx - PCIPassthru: Device 0000:5e:00.0 barIndex 3 type 3 realaddr 0x33800000000 size 33554432 flags 12
YYYY-MM-DDTHH:MM:SS In(05) vmx - PCIPassthru: Device has PCI Express Cap Version 2(size 60)
YYYY-MM-DDTHH:MM:SS In(05) vmx - PCIPassthru: Registered a PCI device for 0000:5e:00.0 vIRQ 0x11, physical MSI = Enabled (vmmInt = Enabled),
IntrPin = 1
YYYY-MM-DDTHH:MM:SS In(05) vmx - PCIPassthru: total number of pages needed (8400896) exceeds limit (5111808), failing
YYYY-MM-DDTHH:MM:SS In(05) vmx - Module 'DevicePowerOn' power on failed.
YYYY-MM-DDTHH:MM:SS In(05) vmx - VMX_PowerOn: ModuleTable_PowerOn = 0
YYYY-MM-DDTHH:MM:SS In(05) svga - SVGA thread is exiting the main loop
YYYY-MM-DDTHH:MM:SS In(05) vmx - Destroying virtual dev for scsi0:0 vscsi=9048770243207189

This issue usually happens if there is not enough allocated MMIO space to the VM for all the GPUs.

Resolution

To solve the issue, it is required to set/configure MMIO (Memory-mapped I/O) parameters for the VM.
Right-click the VM > edit settings, choose VM Options > Advanced > Edit configuration
Once on the Configuration parameters screen, add two more parameters:

pciPassthru.use64bitMMIO = TRUE
pciPassthru.64bitMMIOSizeGB = "MMIO_Space_In_GB"

Example:

pciPassthru.use64bitMMIO = TRUE
pciPassthru.64bitMMIOSizeGB = 128

Hit OK
Then reload the vmx into inventory using Reloading a vmx file without removing the virtual machine from inventory

Below are the steps to determine the value of "MMIO_Space_In_GB" for the parameter "pciPassthru.64bitMMIOSizeGB":

Set up memory-mapped I/O (MMIO) to map system memory to the GPU’s framebuffer memory so that the CPU can pass data to the GPU.
The 64bitMMIOSizeGB value is calculated by adding up the total GB of frame-buffer memory on all GPUs attached to the VM. If the total GPU framebuffer memory falls on a power-of-2, setting pciPassthru.64bitMMIOSizeGB to the next power of 2 works.
If the total GPU frame-buffer memory falls between two powers-of-2, round up to the next power of 2, then round up again, to get a working setting.

Powers of 2 are 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024

For example, if there is one NVIDIA A100 card with 80GB (in between 64GB and 128GB), round up to the next power of 2 (128GB), then round up again to the next power of 2 after that (256GB) to get the correct setting.
Successfully power on all impacted VMs after adjusting pciPassthru.64bitMMIOSizeGB = 256 for a 80GB GPU

Here are some other useful configurations:

2 x 16GB NVIDIA V100 = 32GB, 32 is a power of 2, so round up to the next power of 2 which is 64, set pciPassthru.64bitMMIOSizeGB = 64 to boot.
2 x 24GB NVIDIA P40 = 48GB, which is in-between 32 and 64, round up to 64 and again to 128, requires pciPassthru.64bitMMIOSizeGB = 128 to boot.
8 x 16GB NVIDIA V100 = 128GB, 128 is a power of 2, so round up to the next power of 2 which is 256, set pciPassthru.64bitMMIOSizeGB = 256 to boot.
10 x 16GB NVIDIA V100 = 160GB, which is in-between 128 and 256, round up to 256 and again to 512, set pciPassthru.64bitMMIOSizeGB = 512 to boot.

Additional Information

Reference Blog: https://blogs.vmware.com/apps/2018/10/how-to-enable-nvidia-v100-gpu-in-passthrough-mode-on-vsphere-for-machine-learning-and-other-hpc-workloads.html
Incorrect value in parameter "pciPassthru.64bitMMIOSizeGB" may also fail to power on the VM, and the below errors can be seen in vmware.log:

Msg_post: Error

[msg.efi.pciMmioError] The Firmware could not allocate 134250496 KB of PCI MMIO Increase the size of PCI MMIO and try again