DRS migrating virtual machines to specific ESXi hosts causing cpu or memory utilization to reach upto 100%

Products

• VMware vCenter Server

Issue/Introduction

• DRS is set to fully automated
• DRS is migrating most of the VM's in the cluster to one or two ESXi hosts causing the cpu or memory utilization to reach upto 100%
• Other ESXi hosts in the cluster have a low cpu or memory utilization but DRS still migrates the VM's to only the affected ESXi host
• There are no VM to ESXi host DRS affinity rules
• In the var/log/vmware/vpxd/vpxd.log on the vCenter, you may see entries similar to:

info vpxd[06357] [Originator@6876 sub=cdrsPlmt opID=CdrsLoadBalancer-332f3ab8] Vm [vim.VirtualMachine:vm-####,ESXihost] failed constraint check false on host [ESXihost] with <obj xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:vim25" versionId="8.0.3.0" xsi:type="LocalizedMethodFault"><fault xsi:type="InsufficientAgentVmsDeployed"><hostName>ESXihost</hostName><requiredNumAgentVms>1</requiredNumAgentVms><currentNumAgentVms>0</currentNumAgentVms></fault><localizedMessage></localizedMessage></obj>

• NSX-T is being used in the environment with Malware Prevention Service

Environment

VMware vCenter Server 8.x

Cause

• DRS performs a constraint check for each ESXi host in the cluster and only the ESXi hosts which pass the constraint check are considered for migration of virtual machines. 
• In an environment which uses Malware prevention service with NSX-T, Service virtual machines(SVM) are deployed on every ESXi host in the cluster.
• This issue occurs when the DRS constraint check fail on multiple ESXi hosts in the cluster due to the state of Service virtual machines(SVM).
• As per the error, DRS expects each ESXi host to have one Agent VM(<requiredNumAgentVms>1</requiredNumAgentVms>) but does not find any(<currentNumAgentVms>0</currentNumAgentVms>). Hence constaint check fails with InsufficientAgentVmsDeployed

Resolution

To resolve this issue, check the status of NSX-T SVM's and remediate if any issues found.

Validate the following related to NSX-T SVM's:

• NSX SVM's on each ESXi host in the cluster should be in a powered on state
• In NSX-T manager, vCenter connectivity should be healthy
• Check for alarms related to Malware Prevention Service in NSX UI and take remediation steps as per NSX Malware Prevention Service (MPS) Alarms and Resolutions