vGPU Power-on Failure with libnvidia-vgx.so on NVIDIA H20 GPUs

Products

• VMware vSphere ESXi

Issue/Introduction

A Virtual Machine (VM) configured with an NVIDIA H20 vGPU profile fails to power on. The following error is displayed in the vCenter Server task console:

could not initialize plugin 'libnvidia-vgx.so' for vGPU 'nvidia_h20x-141c'. Failed to start the virtual machine. Module PCIPluginLate power on failed

Logs

The following error messages are recorded in the vmware.log file of the affected VM:

YYYY-MM-DDTHH:MM:SSZ Er(02) vmx - vmiop_log: (0x0): Failed to alloc guest FB memory
YYYY-MM-DDTHH:MM:SSZ Er(02) vmx - vmiop_log: (0x0): Failed to initialize guest FB data for inst 0 with error 2 (Allocating framebuffer for guest failed)

Environment

vSphere ESXi 

Cause

This is a known issue specific to the NVIDIA H20 GPU series. The NVIDIA plugin libnvidia-vgx.so fails to allocate guest framebuffer (FB) memory for certain vGPU profiles when the large system memory configuration is not explicitly enabled for the device.

Resolution

To resolve this issue, apply the following configuration parameters to the Virtual Machine:

Step 1: Enable Large System Memory Support

1. Power off the Virtual Machine.
2. Right-click the VM and select Edit Settings.
3. Navigate to VM Options > Advanced > Edit Configuration.
4. Click Add Configuration Params and enter the following:
   • Name: pciPassthru0.cfg.enable_large_sys_mem
   • Value: 1 (Note: If the GPU is at a different index, replace pciPassthru0 with the appropriate index, e.g., pciPassthru1).

Step 2: Configure 64-bit MMIO

Ensure that the VM is configured with sufficient 64-bit MMIO space for the vGPU profile (e.g., 256 GB or 512 GB for a 141 GB profile):

1. In the same Configuration Parameters window, verify or add:
   • pciPassthru.use64bitMMIO = TRUE
   • pciPassthru.64bitMMIOSizeGB = 256 (or higher as required by the profile).
2. Click OK to save the changes.
3. Power on the Virtual Machine.