Unable to power on GPU based VM on esxi

Products

• VMware vSphere ESXi

Issue/Introduction

Error message received while powering on VM -- "'Could not initialize plugin 'libnvidia-vgx.so' for vGPU 'nvidia_<NVIDIA_PROFILENAME>'. Failed to start the virtual machine. Module PCIPluginLate power on failed."

*Reboot of host will not help.
*VMs will power ON, on certain hosts in cluster but not all despite enough GPU resources.

The default GPU mode is set configured to Shared Direct (vGPU) already.

vmware.log >>

 Er(02) vmx - vmiop_log: (0x0): Failed to alloc guest FB memory
 Er(02) vmx - vmiop_log: (0x0): init_device_instance failed for inst 0 with error 2 (vmiop-display: error allocating framebuffer)
 Er(02) vmx - vmiop_log: (0x0): Initialization: init_device_instance failed error 2
 Er(02) vmx - vmiop_log: display_init failed for inst: 0
 Er(02) vmx - VMIOP: Plugin vmiop-display initialization failed: 2
 In(05) vmx - [msg.vmx.plugin.vmiop.vgpu.failed] Could not initialize plugin 'libnvidia-vgx.so' for vGPU 'nvidia_<NVIDIA_PROFILENAME>'.
 In(05) vmx - Module 'PCIPluginLate' power on failed.
 In(05) vmx - VMX_PowerOn: ModuleTable_PowerOn = 0
 In(05) vmx - Device Interface (pciPassthru0) powering off.
 In(05) vmx - DeviceIfPowerOff: indicating asyncIOThread to exit.
 In(05) vmx - MKSThread: Requesting MKS exit
 In(05) vmx - Stopping MKS/SVGA threads
 In(05) svga - MKSThread: SVGA thread is skipping the main loop
 In(05) vmx - MKS/SVGA threads are stopped
 In(05) mks - MKSRenderMain: Stopping BasicOps
 In(05) mks - MKSRenderMain: Stopped BasicOps
 In(05) mks - MKS PowerOff

 

VM edit setting--

 

Upon running nividia-smi, we see that there is sufficient memory but still the VM will not power on.

 

 

 

 

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

It's a third party issue.

Resolution

Contact Nvidia team to check for any GPU family mismatch and the driver version.