vSphere ESXi 8.0 - vGPU support for 128+ vCPUs (Linux Guest)

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

vCPU and vGPU supported with a VM
ESXi Version	vCPU	vGPU	Supported
7.0	0 - 127	Enabled	Yes
	128+	Enabled	No
8.0	0 - 127	Enabled	Yes
	128+	Enabled	Yes

If a Virtual Machine has more than 127 vCPUs, it must use IOMMU interrupt remapping.  ESX currently doesn't fully support enabling a virtual IOMMU in a Virtual Machine that is also using vGPU. ESXi 7.x and older versions will fail to Power On Virtual Machines, with the following message:

"The virtual machine cannot be powered on because IOMMU virtualization is not allowed with VGPU. To power on the Virtual Machine either disable the virtual IOMMU or remove the VGPU device(s)."

Resolution

ESXi 8.0 has partial support for virtual IOMMU in conjunction with SR-IOV based vGPU configurations.  This support is not available with older non-SR-IOV based vGPU implementations.  Also, when used in this way, DMA remapping is not supported by the virtual IOMMU. 
Furthermore, when adjusting the vCPU count above 128, the UI will automatically enable virtualized "I/O MMU" support. This will create an invalid configuration if a vGPU is attached to the VM, so a VI Admin can enable 128+ vCPUs as follows:

1. Unregister the Virtual Machine (VM)
2. SSH into the host and manually edit the VMX file to set the desired number of vCPUs:

  numvcpus="128"

3. Add the following Virtual Machine configuration parameters:
 
For AMD CPU:

  amd.iommu.supportsPcip=”TRUE”
  vvtd.enable=“TRUE”
  pciPassthru.vmiop.allowViommu=”TRUE”
  pciPassthru.vmiop.enableViommu=”TRUE”
 
For Intel CPU:

  vvtd.enable=“TRUE”
  pciPassthru.vmiop.allowViommu=”TRUE”
  pciPassthru.vmiop.enableViommu=”TRUE”
 
4. Save the VMX file
5. Re-register the VM

In the Linux Guest OS the IOMMU must be in passthrough mode, by setting kernel option “iommu=pt” to disable DMA remapping.

If there are any future changes to the VM configuration:

a1. Unregister the VM
b1. Manually edit the VMX for the desired VM configuration change
c1. Re-register the VM

OR

a2. Unregister the VM
b2. Edit the VMX file to set:

     numvcpus="127"
     vvtd.enable="FALSE"

c2. Re-register the VM
d2. Make VM configuration changes in the UI
e2. Unregister the VM again
f2. Go to Step 2. above to reconfigure the number of vCPUs and re-enable the virtual I/O MMU support.

Additional Information

We were successful in applying the below changes on another VM vmx file
vvtd.enable=TRUE
pciPassthru.vmiop.allowViommu=TRUE
pciPassthru.vmiop.enableViommu=TRUE
Pass thru MMIO size was increased  to 256MB since it was too low.
Set kernel option iommu=pt to disable DMA remapping