vCPU and vGPU supported with a VM | |||
ESXi Version | vCPU | vGPU | Supported |
7.0 | 0 - 127 | Enabled | Yes |
128+ | Enabled | No | |
8.0 | 0 - 127 | Enabled | Yes |
128+ | Enabled | Yes |
If a Virtual Machine has more than 127 vCPUs, it must use IOMMU interrupt remapping. ESX currently doesn't fully support enabling a virtual IOMMU in a Virtual Machine that is also using vGPU. ESXi 7.x and older versions will fail to Power On Virtual Machines, with the following message:
"The virtual machine cannot be powered on because IOMMU virtualization is not allowed with VGPU. To power on the Virtual Machine either disable the virtual IOMMU or remove the VGPU device(s)
."
ESXi 8.0 has partial support for virtual IOMMU in conjunction with SR-IOV based vGPU configurations. This support is not available with older non-SR-IOV based vGPU implementations. Also, when used in this way, DMA remapping is not supported by the virtual IOMMU.
Furthermore, when adjusting the vCPU count above 128, the UI will automatically enable virtualized "I/O MMU
" support. This will create an invalid configuration if a vGPU is attached to the VM, so a VI Admin can enable 128+ vCPUs as follows:
1. Unregister the Virtual Machine (VM)
2. SSH into the host and manually edit the VMX file to set the desired number of vCPUs:
numvcpus="128"
3. Add the following Virtual Machine configuration parameters:
For AMD CPU:
amd.iommu.supportsPcip=”TRUE”
vvtd.enable=“TRUE”
pciPassthru.vmiop.allowViommu=”TRUE”
pciPassthru.vmiop.enableViommu=”TRUE”
For Intel CPU:
vvtd.enable=“TRUE”
pciPassthru.vmiop.allowViommu=”TRUE”
pciPassthru.vmiop.enableViommu=”TRUE”
4. Save the VMX file
5. Re-register the VM
In the Linux Guest OS the IOMMU must be in passthrough mode, by setting kernel option “iommu=pt
” to disable DMA remapping.
If there are any future changes to the VM configuration:
a1. Unregister the VM
b1. Manually edit the VMX for the desired VM configuration change
c1. Re-register the VM
OR
a2. Unregister the VM
b2. Edit the VMX file to set:
numvcpus="127"
vvtd.enable="FALSE"
c2. Re-register the VM
d2. Make VM configuration changes in the UI
e2. Unregister the VM again
f2. Go to Step 2. above to reconfigure the number of vCPUs and re-enable the virtual I/O MMU support.
We were successful in applying the below changes on another VM vmx file
vvtd.enable=TRUE
pciPassthru.vmiop.allowViommu=TRUE
pciPassthru.vmiop.enableViommu=TRUE
Pass thru MMIO size was increased to 256MB since it was too low.
Set kernel option iommu=pt to disable DMA remapping