AI Services vLLM worker VM fails to power on with error "Module 'DevicePowerOn' power on failed" when using NVIDIA L40S GPU Passthrough

Products

• VMware Tanzu Platform

Issue/Introduction

Deploying the AI Services tile with the vLLM model (GPU) fails during the VM power-on phase. The VM is created but is deleted by Bosh within seconds because it fails to Power On.

The following error messages are observed in the vSphere CPI logs.

[2026-07-29T09:02:55.951667 #94140]  WARN -- [req_id cpi-769980]: Error running task 'Drm.ExecuteVmPowerOnLRO'. Failed with message 'Module DevicePowerOn power on failed. ' and fault message 'Module DevicePowerOn power on failed. ,Failed to start the virtual machine.'.
I, [2026-07-29T09:02:55.952262 #94140]  INFO -- [req_id cpi-769980]: Deleting vm: 445b3b6d-xxxx-xxxx-8xx0-xxxxxxxxxx_genai-models_xxxxxxxxxxx


I, [2026-07-29T09:03:01.322470 #94140]  INFO -- [req_id cpi-769980]: Module DevicePowerOn power on failed.  - /var/vcap/data/packages/vsphere_cpi/16cb942b68564e503eb15035e235a68ed38be3bb/lib/cloud/vsphere/task_runner.rb:30:in `block in run'
/var/vcap/data/packages/vsphere_cpi/16cb942b68564e503eb15035e235a68ed38be3bb/lib/cloud/vsphere/retryer.rb:9:in `block in try'
<internal:numeric>:237:in `times'

 

Environment

Tanzu Platform 10.x

AI Services 

 

Cause

This issue can occur when a virtual machine using Direct PCI Passthrough (DirectPath I/O) for high-end GPUs (such as the NVIDIA L40S or A100) does not have enough Memory-mapped I/O (MMIO) space allocated. vSphere requires specific configuration parameters to map the GPU's large frame-buffer memory into the VM's address space.

 

Resolution

To resolve this, you must configure the 64-bit MMIO settings in the AI Services tile infrastructure configuration.

AI Services tile, add these parameters to the Infrastructure Config:

 

• pciPassthru.use64bitMMIO: Set to TRUE
  
  
• pciPassthru.64bitMMIOSizeGB: "<size>" (e.g., 64 or 128 depending on your GPU VRAM size)

 

After configuring the parameters save the config and run apply changes again selecting the bosh director and AI Service tile.

Now the vllm worker vm will be deployed successfully.

Additional Information

Below are the steps to determine the value of "MMIO_Space_In_GB" for the parameter "pciPassthru.64bitMMIOSizeGB":

• Set up memory-mapped I/O (MMIO) to map system memory to the GPU’s framebuffer memory so that the CPU can pass data to the GPU.
• The 64bitMMIOSizeGB value is calculated by adding up the total GB of frame-buffer memory on all GPUs attached to the VM. If the total GPU framebuffer memory falls on a power-of-2, setting pciPassthru.64bitMMIOSizeGB to the next power of 2 works.
• If the total GPU frame-buffer memory falls between two powers-of-2, round up to the next power of 2, then round up again, to get a working setting.
• For example, if there is one NVIDIA A100 card with 80GB (in between 64GB and 128GB), round up to the next power of 2 (128GB), then round up again to the next power of 2 after that (256GB) to get the correct setting
  
  Powers of 2 are 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024