Unable to vMotion VMs with vGPU to a specific ESXi host, after upgrading the vGPU driver for host and guest OS.

Products

• VMware vSphere ESXi

Issue/Introduction

When you attempt to migrate a VM with vGPU, the migration wizard fails due to compatibility issues such as:

Insufficient resources
One or more devices (pciPassthru0) required by VM <VM_name> are not available on host <Host_name>

You will see an error similar to (some values such as timestamps, opIDs, and VM/Host ID numbers will be different in your environment:

2024-10-15T18:39:46.842+09:00 verbose vpxd[06824] [Originator@6876 sub=OMM opID=########################] AH Evaluating [vim.VirtualMachine:vm-#####,VM_name] pciPassthru0 isRestoring:true predicate tree:
--> (&& (== _moduleName nvidia)
-->     (== _vgpu grid_t4-1q)
-->     (== _deviceMetadata ########################)
-->     (<= _minVersion 1179649)
-->     (>= _maxVersion 1179649))
2024-10-15T18:39:46.842+09:00 verbose vpxd[06824] [Originator@6876 sub=OMM opID=########################] AH No match found
2024-10-15T18:39:46.843+09:00 verbose vpxd[06824] [Originator@6876 sub=Iofilter opID=########################] Disk 2000 has 0 IOFilter(s)
2024-10-15T18:39:46.843+09:00 info vpxd[06824] [Originator@6876 sub=VmCheck opID=########################] CompatCheck results: (vim.vm.check.Result) [
-->    (vim.vm.check.Result) {
-->       vm = 'vim.VirtualMachine:########################:vm-####',
-->       host = 'vim.HostSystem:########################:host-##',
-->       error = (vmodl.MethodFault) [
-->          (vim.fault.InsufficientResourcesFault) {
-->             faultMessage = (vmodl.LocalizableMessage) [
-->                (vmodl.LocalizableMessage) {
-->                   key = "com.vmware.vim.vpxd.vmcheck.assignHwNotAvailable",
-->                   arg = (vmodl.KeyAnyValue) [
-->                      (vmodl.KeyAnyValue) {
-->                         key = "host",
-->                         value = "Host_namel"
-->                      },
-->                      (vmodl.KeyAnyValue) {
-->                         key = "vm",
-->                         value = "VM_name"
-->                      },
-->                      (vmodl.KeyAnyValue) {
-->                         key = "missing",
-->                         value = "pciPassthru0"
-->                      }
-->                   ],
-->                }
-->             ],
-->             msg = ""
-->          }
-->       ],
-->    }
--> ]

Environment

VMware vSphere ESXi 7.x

Cause

The minVersion/maxVersion values for each instance in ah-trees.conf on the problematic host are not set.

/etc/vmware/ah-trees.conf

:
      {
       "description" : "grid_t4-1b",
       "NodeName" : "grid_t4-1b",
       "nodeType" : "INSTANCE",
       "InstanceId" : "~1~nvidia~0000:d8:00.0=grid_t4-1b",
:
        "_minVersion" : {
         "overridable" : false,
         "intValue" : 0   <<<<<
        },
        "_maxVersion" : {
         "overridable" : false,
         "intValue" : 0   <<<<<
        }

:

 

Resolution

Run the following command on the problematic host  to refresh ah-trees.conf file.

# esxcli graphics host refresh