Power on operation fails for the VM configured to use NVIDIA GPU with Error: "The host does not have sufficient memory resources to satisfy the reservation"

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• This KB is specific to below scenario:

• • NVIDIA Device:  NVIDIA Corporation  AD102GL [L40S]
  • Total ESXi host Memory = 512 GB
  • 4 VMs configured to use NVIDIA GPU with Reserved Memory of 128GB each = 4 x 128 GB  = 512 GB

• 3 VMs power on successfully.
• But the power-on operation of 4th VM fails with Error: "The host does not have sufficient memory resources to satisfy the reservation"

• On vCenter server, "Requested Memory" to power-on the VM is more as compared to the "Unreserved Memory" available on ESXi host

vpxd.log

20##-##-07T1##:31:21.686Z info vpxd[05703] [Originator@6876 sub=vpxLro opID=ma0x1hns-75##23-auto-g314-h5:70176271-##-01] [VpxLRO] -
- BEGIN lro-24958829 --  -- DrmClusterVmPowerOn --
20##-##-07T1##:31:21.690Z info vpxd[05703] [Originator@6876 sub=cdrsPlmt opID=ma0x1hns-75##23-auto-g314-h5:70176271-##-01] Vm [vim.VirtualMachine:vm-1##6,VM04] failed constraint check true on host [vim.HostSystem:host-1##3,10.##.##.126] with <obj xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:vim25" versionId="8.0.3.0" xsi:type="LocalizedMethodFault"><fault xsi:type="InsufficientHostMemoryCapacityFault"><faultMessage><key>com.vmware.drs.constraints.mem.InsufficientHostMemoryCapacityFault</key><arg><key>requested</key><value xsi:type="xsd:long">1426####3504</value></arg><arg><key>unreserved</key><value xsi:type="xsd:long">11411###0352</value></arg><arg><key>host</key><value xsi:type="xsd:string">10.##.##.126</value></arg></faultMessage><host type="HostSystem">host-1##3</host><unreserved>114113380352</unreserved><requested>142689173504</requested></fault><localizedMessage></localizedMessage></obj>
 

<unreserved>114113380352</unreserved>  => 106.28 GB

 <requested>142689173504</requested> => 132.89 GB

Environment

VMware vSphere 7.x

VMware vSphere 8.x

NVIDIA Corporation AD102GL [L40S]

Cause

 When "Requested Memory" to power-on the VM is more as compared to the "Unreserved Memory" available on ESXi host, VM fails to power-on.

Resolution

• When the PCI GPU device is assigned to a VM, the virtual machine must have a memory reservation for the full configured memory size for the VM.
• The Memory reservation cannot be reduced.
• You can engage NVIDIA for the recommended PCI GPU Memory size that can be reserved for the VMs using NVIDIA Corporation  AD102GL [L40S].

Additional Information

Using GPUs with Virtual Machines on vSphere – Part 2