vMotion with vGPU fails with "Insufficient resources" during compatibility check on NVIDIA Mixed-Size Mode on A10 VGPU

Products

• VMware vCenter Server

Issue/Introduction

• When attempting to migrate (vMotion) a Virtual Machine (VM) configured with NVIDIA vGPUs, the migration wizard may fail during the compatibility check.
• The migration fails with the error:
  Insufficient resources. One or more devices (pciPassthru0) required by VM are not available on host.

• The destination ESXi host has sufficient free GPU memory and identical hardware models, the compatibility check still fails.
• The VM may fail to power on after a manual move or during a cold migration.

Cause

• The A10 vGPU types, support a maximum combined resolution based on the number of available pixels, which is determined by their frame buffer size.
• The A10 series supports, a maximum of 8 vGPUs per physical GPU when operating in Mixed-Size mode.
• The maximum supported configuration change is due to the structural changes in the Ampere architecture, if the destination host has already reached this partition limit or is utilizing a conflicting vGPU profile type, the system will report that the required pciPassthru0 device is unavailable.

Resolution

This is an expected behavior due to physical resource unavailability on the hardware level.

To resolve or further diagnose this limitation, it is recommended to engage NVIDIA, refer Virtual GPU Software User Guide - NVIDIA Docs

Additional Information

The Mixed-Size mode can be determined by running the following command via SSH using root user to the ESXI host : 

nvidia-smi -q | grep -i "Heterogeneous"

Expected output (for Mixed-Size mode) : 

        vGPU Heterogeneous Mode                        : Enabled

        Heterogeneous Time-Slice Profiles              : Supported

        Heterogeneous Time-Slice Sizes                 : Supported