vGPU enabled VM fails to migrate from ESXi 7.0 U3 to ESXi 8.0U3e or newer with the error "PCIPassthru: Failed to adjust IOMMU for VM: Not supported"

Products

• VMware vSphere ESXi

Issue/Introduction

• The virtual machines (VMs) are configured with vGPU.
• The source ESXi host is running version 7.0 U3.
• The destination ESXi host is upgraded to ESXi 8.0 U3e or later.
• Migration of non-vGPU VMs works without issues.
• Cold migration is possible for vGPU-configured VMs.
• The vmware.log file located  /vmfs/volumes/vm_name/ on these VMs may contain the following entries:
  
  YYYY-MM-DDTHH:MM:SS.653Z In(05) vmx - Received migrate 'from' request for mid id 570#######314, src ip <source_host_IP>.
YYYY-MM-DDTHH:MM:SS.654Z In(05) vmx - MigrateSetInfo: state=MIGRATE_FROM_VMX_INIT srcIp=<source_host_IP> dstIp=<destination_host_IP> mid=570#######314 uuid=5####a-8##a-13##ec-####-#######priority=high
YYYY-MM-DDTHH:MM:SS.446Z In(05) worker-###### - PCIPassthru: Failed to adjust IOMMU for VM: Not supported
YYYY-MM-DDTHH:MM:SS.446Z Cr(01) worker-###### - PANIC: VERIFY bora/devices/pcipassthru/pciPassthru.c:3694
YYYY-MM-DDTHH:MM:SS.446Z In(05) worker-###### - VMIOP: informing the plugin vmiop-display of Panic
YYYY-MM-DDTHH:MM:SS.197Z Wa(03) worker-###### - A core file is available in "/var/core/vmx-zdump.000"

Environment

• vSphere ESXi 7.x
• vSphere ESXi 8.x

Resolution

Broadcom is working towards a permanent fix for this issue.

Workaround

Option1

• Upgrade the ESXi hosts in the cluster from 7.x  to 8.0 Update 3
   
  Note: vMotion of vGPU-configured VMs is  limited between ESXi 7.x and 8.0 Update 3 or earlier(Interim workaround)
  
  
• Once the cluster upgrade to ESXi 8.0 Update 3 is complete, proceed to update the host(s) to later patches if required 
  
  

Option2

• Virtual Machines can be cold-migrated (migrated while powered off) across versions, including from 7.0 U3 to 8.0 U3e or later.
  • Note: Cold migration may result in VM downtime depending on production dependencies.

Additional Information

Impact:

• Not able to migrate VMs after patching/upgrading the ESXi to 8.0U3e which impacts the upgrade process.