Virtual Machine Power-On Fails with vMX Process Crash and GSP Timeout on ESXi 8.0 with NVIDIA L40 GPUs

Products

• VMware vSphere ESXi

Issue/Introduction

On VMware ESXi 8.0 Update 3 hosts equipped with NVIDIA L40 or L4 GPUs using vGPU profiles, the following issues may occur:

• Virtual machines (such as VDI desktops) fail to power on, and management consoles like Omnissa Horizon report a "Disconnected from virtual machine" error.
• The vmware.log for the affected VM records the following errors, leading to a vMX process crash (Signal 11/Panic):
  

  YYYY-MM-DDThh:mm:ss.xxxZ Er(0x) vmx - vmiop_log: (0x0): Timeout waiting for GSP Plugin before triggering doorbell
  YYYY-MM-DDThh:mm:ss.xxxZ Er(0x) vmx - vmiop_log: (0x0): Failed to negotiate CPU <-> GSP version with error:0x7
  YYYY-MM-DDThh:mm:ss.xxxZ Er(0x) vmx - vmiop_log: (0x0): Failed to initialize guest FB data for inst 0 with error 7 (CPU <-> GSP version negotiation failed!)
  YYYY-MM-DDThh:mm:ss.xxxZ Er(0x) vmx - vmiop_log: (0x0): Initialization: guest FB dependent init failed error 7
  YYYY-MM-DDThh:mm:ss.xxxZ[+1.670] Wa(03) vthread-xxxxxxx - Caught signal 11 -- tid xxxxxxx

Environment

VMware vSphere 8.x

Cause

The cause of the power-on failure is a timeout during the vGPU initialization process because the NVIDIA GSP (GPU System Processor) failed to respond to requests from the ESXi host.

Resolution

Please contact your hardware vendor to find out why timeout during the vGPU initialization process.