vGPU Power-on Failure "Module DeviceGroupLate power on failed" despite available resources on NVIDIA H200 GPUs

Products

• VMware vSphere ESXi

Issue/Introduction

When attempting to power on a Virtual Machine (VM) or Tanzu Kubernetes Grid (TKG) worker node configured with NVIDIA H200 vGPU profiles, the task fails.

• In vCenter, the error message Module 'DeviceGroupLate' power on failed or GenericVmConfigFault is displayed.
• The ESXi host nvidia-smi command shows that physical GPU capacity and vGPU partitions are available.
• The VM fails to initialize the NVSwitch device group.

• Symptoms/Logs: 
  In the VM's vmware.log  
  VMIOP: GPU manager activate FM partition id X failed: -7.
VMIOP: Failed to initialize NVSwitch device group.
Module 'DeviceGroupLate' power on failed.

  ESXi host vmkernel.log:
  Admission failure in path: host/vim/vmvisor/gpuManager:gpuManager.xxxx:pt.xxxx
requires 4 KB, asked 4 KB from gpuManager which has 40960 KB occupied and 0 KB available.

Environment

VMware vSphere ESXi

Cause

The ESXi host's gpuManager service (GPU Management Service) encounters a resource accounting hang or a metadata discrepancy. Even if the physical GPU is idle, the gpuManager service incorrectly reports "0 KB available" to the ESXi resource scheduler, leading to an admission failure for the VM's vGPU request.

Resolution

To resolve this issue without rebooting the entire ESXi host, restart the GPU management service to reset the resource counters:

1. Enable SSH on the affected ESXi host.
2. Login as root.
3. Restart the gpuManager service:  /etc/init.d/gpuManager restart
    
4. Wait approximately 30 seconds for the service to re-initialize and sync with the NVIDIA Fabric Manager.
5. Re-attempt the VM power-on task.