VM failed to power on with error "Could not initialize plugin 'libnvidia-vgx.so' for vGPU 'nvidia_l4-4q'. Failed to start the virtual machine. Module 'PCIPluginLate' power on failed."

Products

• VMware vSphere ESXi

Issue/Introduction

NVIDIA VGPU VM fails to power on ESXi 8.0 Update 2 when using NVIDIA L4, with the below error message:

"Could not initialize plugin 'libnvidia-vgx.so' for vGPU 'nvidia_l4-4q'. Failed to start the virtual machine. Module 'PCIPluginLate' power on failed."

Symptoms:

• Power-on VM fails with attached vGPU device "nvidia_l4-4q" on ESXi 8.0 Update 2
• If vm is powered on without attaching vGPU device, vm successfully powers on.
• If GPU is already on Shared direct mode, and it has plenty of resources available..
• When nvidia display mode is in use and disabling does not seem to be an option
• hostd and vmware.log shows below errors:

Hostd:

[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[133728744]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/########-########-####-############/<vm_name>/vm_name.vmx]` `Power On message: Could not initialize plugin 'libnvidia-vgx.so' for vGPU 'nvidia_l4-4q'.  
[YYYY-MM-DDTHH:MM:SS]Z Db(167) Hostd[133728701]: -->` `Module 'PCIPluginLate' power on failed``. 
[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[133728701]: --> Failed to start the virtual machine.

vmware.log: 
[YYYY-MM-DDTHH:MM:SS] In(05)+ vmx - Power on failure messages: Could not initialize plugin 'libnvidia-vgx.so' for vGPU 'nvidia_l4-4q'.  
[YYYY-MM-DDTHH:MM:SS] In(05)+ vmx - Module 'PCIPluginLate' power on failed.  
[YYYY-MM-DDTHH:MM:SS] In(05)+ vmx - Failed to start the virtual machine.

Environment

VMware vSphere ESXi 8.0.x

Cause

Issue with NVIDIA L4 GPU driver in use

Resolution

Updating the driver to the latest Long-Term Support (LTS) version 16.9 successfully resolved the issue.

Additional Information

Alternatively, disabling the display mode can be considered (If Required/Applicable)

Steps To Disable Display Mode:

• Install NVIDIA Display Mode Selector Tool on the VM (guest OS)
• Run this CMD "displaymodeselector --gpumode physical_display_disabled"
• Reboot guest OS.