Powering off ESX agent VM cause all the regular VMs to migrate out of the host in a DRS enabled cluster.

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• Powering off ESX agent VM cause all the regular VMs to migrate out of the host in a DRS enabled cluster.
• In vCenter's vpxd logs, you will see vMotion related errors for other VMs in the cluster.
• Logs will also indicate "currentNumAgentVms" count as "0":
  /var/log/vmware/vpxd/vpxd.log:
  2023-01-04T07:12:11.962Z info vpxd[06256] [Originator@6876 sub=cdrsPlmt opID=CdrsLoadBalancer-60b7acd9] Vm [vim.VirtualMachine:vm-8029,<vm-name>] failed constraint check false on host [vim.HostSystem:host-28,<ESXi-hostname>l] with <obj xmlns:xsd=http://www.w3.org/2001/XMLSchema xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xmlns="urn:vim25" versionId="7.0.3.0" xsi:type="LocalizedMethodFault"><fault xsi:type="InsufficientAgentVmsDeployed"><hostName><ESXi-hostname></hostName><requiredNumAgentVms>1</requiredNumAgentVms><currentNumAgentVms>0</currentNumAgentVms></fault><localizedMessage></localizedMessage></obj>
• Same logs were observed while VMs were unable to power on, when being powered on through vCenter/vSphere Web Client:
  

Environment

• VMware vSphere ESXi 7.0

Cause

• This issue occurs when number of powered on ESXi agent VMs in DRS enabled cluster is less than the required number of powered on ESXi agent VMs. 
• DRS cluster advance option "numAgentVmsRequired" indicates the required number of ESXi agent VMs powered on a host within a DRS enabled cluster and option is set by EAM service. When set, DRS will ensure that all hosts has the required number of ESXi agent VM powered on and if not DRS load balancer will gradually migrate the regular VMs from this host to other hosts in cluster which has the required number of ESXi agent vm and also DRS will not allow power-on or migration of regular VM into the host that does not have the required number of ESXi agent VMs.

Note - Manually Changing this advanced option "numAgentVmsRequired" has no effect as this option is controlled by EAM service.

Resolution

• ESX agent VM deployed on the host within a DRS enabled cluster should not be powered off by a user.

Additional Information

• For VM power on issue, restart of eam service on the vCenter server resolved this: 
  
  1. SSH to the vCenter server as root
  2. Switch the console to bash:
     > shell
  3. Restart service eam:
     service-control --restart vmware-eam