Virtual Machine with NVIDIA shared vGPU fails to boot when deployed by Aria Automation

Products

• VMware vSphere ESXi

Issue/Introduction

• Nvidia graphics VIB installed on esxi host.
• PCI passthrough for the GPU is enabled on the vSphere Client Host Configure tab, expand Hardware and click PCI Devices.
  
  
• You configure a passthrough PCI device on a virtual machine in the vSphere Client.
  
  
• nvidia-smi does not run on the host.
  
  
• nvidia kernel module does not load at boot
  
  
• Older vGPU Software Release is installed on the esxi host

Environment

esx 8.x

Cause

Misconfiguration on the server hardware bios, esx host and virtual machine.

Note, this is not an Aria Automation Deployment issue.

Resolution

• Check with server hardware vendor to enable required BIOS settings for GPU passthrough.
  
  
  • Enable SR-IOV Global Enable to Enable
    System BIOS ==> Integrated Devices ==> SR-IOV Global Enable. This option is set to Disabled by default. Save BIOS changes before exit.
•  
• Disable Passthrough mode for the vGPU in esxi configuration "Manage" > Hardware" > PCI Devices" and enable SR-IOV mode.

• Install Active vGPU Software Release. 
  
  
  • Confirm nvidia vmkernel module is loaded
    
    #  vmkload_mod -l | grep nvidia

  • To list the VIBs installed on the ESXi host and ensure that the NVIDIA VIB was done correctly, use the command:

    # esxcli software vib list |grep –i NV

  • To confirm that the GPU card and ESXi are working together correctly, use the command:

    # nvidia-smi

    

• Configure the Virtual Machine to use the NVIDIA shared GPU.

Reference Step-by-Step Installation of NVIDIA Virtual GPU Software on VMware vSphere

Additional Information

• Details on setting up the NVIDIA Virtual GPU (vGPU) technology for GPUs on vSphere
  
  
• Installing and configuring the NVIDIA VIB on ESXi