DRS "Should run on hosts in group" Affinity Rules are not respected, leading to an overload of an ESXi host

Products

• VMware vCenter Server

Issue/Introduction

Consider this scenario:

There is a Cluster with 4 ESXi hosts. These are spread over 2 Sites with 2 ESXi on each Site.
There are VMs hosted only on one Site which is enforced through a MUST RUN Rule.
And there are VMs that could be hosted on both Sites so they can switch the Site with a SHOULD RUN Rule.
If a ESXi is set to maintenance Mode all VMs on SIte 1 will only migrated to the second ESXi on Side 1.

That host then experiences 99% CPU and 99% RAM and VMs will not reachable.

If the SHOULD RUN Rule is disabled... VMs will be migrated to all of the other 3 ESXi and the cluster is balanced.

Environment

vSphere 8.0

vSphere 7.0

Cause

by default DRS weighs active memory (the memory the VM consumes currently) higher than consumed memory (the maximum the VM has ever consumed) and allocates accordingly. 

Resolution

Set PercentIdleMBInMemDemand in the advanced cluster settings to 100

Explanation:

The weight that is given to consumed vs active memory is adjusted by the PercentIdleMBInMemDemand parameter in the advanced cluster settings. It defaults to 25(%) where 0 is only active memory being considered and 100 where only consumed memory is considered. This overrides the preference for the local ESXi host that SHOULD RUN induces

Additional Information

for further reading see

Load Balancing vSphere Clusters with DRS

Tuning the DPM Algorithm