DRS does not balance workloads in mixed GPU and non-GPU clusters

Products

• VMware vCenter Server

Issue/Introduction

In vSphere clusters containing both GPU-enabled hosts and standard (non-GPU) hosts, DRS may fail to migrate non-GPU virtual machines (VMs) to GPU-enabled hosts, even if those hosts appear to have low memory consumption. This results in an uneven distribution of computational load across the cluster.

• Non-GPU VMs (such as SAP instances) are immediately migrated back to non-GPU hosts after a manual vMotion to a GPU host.
• vSphere UI shows low "Consumed Memory" on GPU hosts, but DRS reports high memory pressure.
• The cluster contains hosts running Kubernetes/TKG workloads with vGPU best-effort configurations.
• In the vCenter, you may see "Unable to apply DRS resource settings" alarms.

Environment

VMware vSphere

Cause

The DRS algorithm utilizes "Memory Demand" rather than "Consumed Memory" to make migration decisions.

In GPU environments, specifically with vGPU best-effort VMs, the InitialSwapReservation (vmxSwappedOverhead) can be significantly inflated to cover the framebuffer size of the GPU devices.

This overhead is added to the total memory demand, often causing GPU hosts to exceed 100% memory demand even when actual VM usage is low.

Resolution

This is expected behaviour.

To avoid this issue, recommendation is to keep ESXi host clusters homogenous from the hardware perspective.