vGPU VM placement may result in sub-optimal profile capacity due to Host/GPU distribution

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
VM PowerOn results in "Insufficient resources. One or more devices (pciPassthru...) required by VM ... are not available on host"

Environment

VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 7.x

Cause

DRS will distribute vGPU VMs in a breadth-first manner across a cluster's hosts. Fractional vGPU profile allocation for a VM may be subject to homogeneous profile mutual exclusion rules.

Resolution

With vCenter 8.0 U2 and vSphere 8.0 U2, Cluster level consolidation of vGPU VMs with only 1:1 (whole) profiles was added.
Instead of a breadth first distribution amongst Cluster Hosts, DRS will attempt a depth first distribution. This functionality is an OPT-IN using the following DRS Cluster Advanced Option:

  Option: VgpuVmConsolidation
  Value: 1

Workaround:

To workaround the issue, please follow the below mentioned steps:

• Manually migrate vGPU VMs to a desired host to open up unused Physical GPU capacity.
• Use the same vGPU profile configuration in all vGPU VMs in a Cluster.
• Enable Host "GPU Consolidation", please refer Configuring Host Graphics for more information.
• If "DRS Automation" is active, consider putting the Cluster or VM into "Partially Automated" mode, please refer Creating a DRS Cluster for more information.