vGPU Virtual Machine automated migration for Host Maintenance Mode in a DRS Cluster

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

When an ESXi host that is a member of a DRS cluster is switched into Maintance Mode, DRS can utilize vMotion in order to migrate virtual machines to other compatible hosts in the cluster.
However, if there are virtual machines with a vGPU device configured running on the host, the Enter Maintenance Mode task might fail with an error:

DRS failed to generate a vMotion recommendation for a virtual machine on a host entering Maintenance Mode.
In addition, when reviewing the DRS recommendations in <Cluster> → Monitor → DRS → Recommendations, there might be one or more migration recommendations for vGPU VMs, with the reason given as:

Destination host is selected for fixing policy/rule violation or its healthy state is better.
Manually migrating the vGPU VMs to other hosts works fine.

Environment

VMware vCenter Server 7.0.x / 8.0.x
VMware vCenter 9.x
VMware vSphere ESXi 7.0.x / 8.0.x
VMware vSphere ESX 9.x
Nvidia vGPU

Cause

Due to the underlying architecture, migrating virtual machines with vGPU configured will lead to considerably increased VM stun times, usually longer than the default stun timeout of 100 seconds.
Therefore DRS by default handles vGPU VMs in semi-automated mode and will only give recommendations rather than migrating those virtual machines automatically.

Resolution

Starting with vSphere 8.0 U2, DRS can estimate the Stun Time for a given vGPU VM configuration. When the DRS Cluster Advanced Options are set and the Estimated VM Devices Stun Time for a VM is lower than the VM Devices vMotion Stun Time limit, DRS will automatically migrate the VM.

To enable this functionality, make sure that the infrastructure meets the following requirements:

vSphere Cluster Services are in a healthy state. Refer to vSphere Cluster Services (vCLS) Known Issues/Corner Cases
Configuration of the VM's vGPU devices through the vSphere UI (vSphere Client) only
A dedicated vMotion network is configured and healthy (Example: vMotion NICs have been set up during Cluster QuickStart)
Healthy GPU and GPU driver (e.g. no Xids or Assertions).

Once this was confirmed, add the following DRS Cluster Advanced Options:

Option	Value
`PassthroughDrsAutomation`	`1`
`LBMaxVmotionPerHost`	`1`

For vGPU VMs with Stun Times exceeding the vMotion Stun Time Limit (default 100 seconds), a VI Admin can add the following DRS Cluster Advanced Option:

Option	Value
`VmDevicesStunTimeTolerated`	<number of seconds, greater than any VMs Estimated Stun Time in the Cluster> (Default 100 seconds)

Workaround for vCenter versions prior to 8.0 Update 2

Starting with vCenter Server 7.0 Update 3f, a DRS Cluster Advanced Option was introduced as a way to allow virtual infrastructure admins to override the default DRS behavior and instead to OPT-IN to automated evacuation of vGPU VMs. To enable this functionality, configure the following option:

Option	Value
`VgpuMMAutomationTimeoutSecs`	`-1`

This override comes with the following behavior changes:

Evacuation of vGPU Virtual Machines is automated, but remains subject to the general 100-second vMotion timeout.
During Switchover a vGPU Virtual Machines Stun Time may exceed 10 seconds (dependent on both network bandwidth and the size of the vGPU profile).
Evacuation of Virtual Machines is serialized to avoid network contention.

Requirements:

Extra vGPU capacity will be needed in the DRS cluster (Example: duplicate host configuration for the host going into Maintenance Mode)
No compatibility issues were reported for the VMs on the host going into Maintenance Mode.

For older releases, manually evacuate any running vGPU virtual machines prior to setting the host into maintenance mode. If VM placement issues arise, reduce the DRS automation level to "Partially automated". Refer to Configure Virtual Graphics on vSphere for more information.