vGPU VM fails to power on with "DevicePowerOnEarly" error and NVIDIA "GPU is lost" status

Products

• VMware vSphere ESXi

Issue/Introduction

• A Virtual Machine configured with a vGPU profile (e.g., nvidia_l20_48q) fails to power on.
• vCenter Task and Events report: Module "DevicePowerOnEarly" power on failed and Insufficient graphics resources.
• The ESXi host vmkernel.log contains: WARNING: PCI: Dev 0000:xx:00.x is unresponsive after reset 
• Running nvidia-smi on the host returns: Unable to determine the device handle for GPU0: GPU is lost. Reboot the system to recover this GPU.

Environment

VMware vSphere ESXi

Cause

The physical NVIDIA GPU has entered an error state (GPU Lost) due to a PCIe bus completion timeout during a reset operation. This causes the hardware to become unresponsive to the driver and the ESXi hypervisor, resulting in the loss of all associated graphics resources for the vSphere resource scheduler.

Resolution

1. Perform a Cold Power Cycle: Shut down the affected ESXi host and perform a full "Cold Boot" (unplug power cables or use the BMC/iDRAC/ILO to perform a full power reset). This is required to clear the PCIe register lockup state.
2. Verify Hardware Recognition: After the host reboots, run nvidia-smi to ensure the GPU is visible and the "GPU is lost" error is cleared.
3. Check for Hardware Faults: If the error persists after a cold boot, inspect the physical hardware, reseat the GPU, or consider hardware replacement as the card may be physically damaged.