Error: "Could not initialize plugin /usr/lib64/vmware/plugin/libnvidia-vgx.so" when powering on NVIDIA vGPU VMs

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual Machines configured with NVIDIA vGPU fail to power on. This issue is typically caused by SR-IOV (Single Root I/O Virtualization) being disabled in the server BIOS.
  
  

• SSH into the ESXi host and check /var/run/log/vmkernel.log: Explicit warnings indicate the BIOS is blocking Virtual Functions (VFs):

  WARNING: PCI: 1615: Enable VFs failed... Please make sure the system has proper BIOS installed and enabled for SRIOV.
NVRM: SRIOV may be disabled

• Within the same SSH session, check /var/log/hostd.log: The license manager reports sriov = 0, confirming the feature is unavailable to the hypervisor:

  LicFeature ... sriov = 0, ... vgpu = 0


• SSH into esxi and run the below command:

  lspci -v | grep -A 5 "NVIDIA" | grep 'Virtual Function'

  Observation: The output only shows the physical controller, confirming the hardware is not exposing Virtual Functions.

Cause

SR-IOV (Single Root I/O Virtualization) is disabled in the server BIOS. Although the NVIDIA driver is successfully loaded, the hardware prevents it from initializing the necessary Virtual Functions (VFs). ESXi requires SR-IOV to be enabled at the hardware level to successfully map and utilize vGPU resources.

Resolution

Enable virtualization support at the hardware level and verify the configuration:

1. Reboot the physical server and enter the BIOS/UEFI settings.
   
   
2. Enable SR-IOV
   
   Note: Consult your hardware vendor's documentation for the specific menu location of these settings.
   
   
3. Save the settings and reboot the server.
   
   
4. Once the ESXi host is back online, verify the fix by running this command via SSH to the ESXi host:
   

   lspci -v | grep -A 5 "NVIDIA" | grep 'Virtual Function'

   Confirm the output contains "Virtual Function" entries, indicating that SR-IOV is active

5. Power on the VM.

Additional Information

If the logs do not show the SR-IOV errors above, the issue may be related to an incompatible or outdated driver. Refer to this article for standard driver troubleshooting:

• Installing and configuring the NVIDIA VIB on ESXi - Follow this if a driver upgrade is required.