File Service upgrade causes all VMs to vMotion to the same host

Products

• VMware vSAN

Issue/Introduction

• When vSAN file service upgrade finishes you will notice that VMs will start migrating to the same host
• In vCenter vpxd logs (/var/log/vmware/vpxd) we will see this InsufficientAgentVmsDeployed messages for all the hosts in the cluster where VMs are moving out of:
  

  vpxd[#####] [Originator@#### sub=###### opID=<Op-ID>-########] Vm [vim.VirtualMachine:vm-#####,<VM-Name>] failed constraint check false on host [vim.HostSystem:host-#####,<Host-Name>] with versionId="8.0.3.0" xsi:type="LocalizedMethodFault"><fault xsi:type="InsufficientAgentVmsDeployed"><hostName><Host-Name></hostName><requiredNumAgentVms>1</requiredNumAgentVms><currentNumAgentVms>0</currentNumAgentVms></fault><localizedMessage></localizedMessage></obj>

• Vsanmgmt logs (/var/log/vmware/vsan-health/vmware-vsan-health-service.log) will show the only host as healthy and all the other hosts in the cluster as unhealthy:
  

  vsan-mgmt[#####] [VsanEamUtil::_WaitForAgencyProgress opID=###-#######-#########-#######] EAM: Agent 'eam.Agent:########-####-####-####-############': Runtime = {
     status = green,
     host = 'vim.HostSystem:host-####',
     issue = (eam.issue.Issue) [],
     vmHook = None

Environment

8.x

 

Resolution

Workaround:

Put DRS in manual mode for the duration of the time when upgrading the vSAN File Service.

 

Note: 

In clusters with a large number of ESXi hosts, avoid switching DRS to Automated until all vSAN File Service VMs have been successfully recreated. In some cases, the creation of these VMs may take longer than expected.

You can verify their status by checking the Events tab and confirming the most recent time each vSAN File Service VM was powered on.