vCLS Virtual Machines are not powered on after vSAN cluster shutdown and start.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

The below log is found at /var/log/vmware/eam/eam.log:

ERROR | cluster-agent-3 | AuditedJob.java | 106 | JOB FAILED: [#440855571] InstallClusterAgentJob(ClusterAgent(ID: 'Agent:3861f614-b6e3-4ae1-a9c8-0f1b321157a3:null'))
com.vmware.eam.agent.crud.PreCheckException: Host HostSystem:host-1009 not fully operational. ClusterAgent(ID: 'Agent:3861f614-b6e3-4ae1-a9c8-0f1b321157a3:null')
 at com.vmware.eam.agent.crud.impl.PreCheckBase.wrapHostNotOperational(PreCheckBase.java:386) ~[eam-server.jar:?]
 at com.vmware.eam.agent.crud.impl.PreCheckBase.vmHostInstallCheck(PreCheckBase.java:330) ~[eam-server.jar:?]
 at com.vmware.eam.agent.crud.impl.PreCheckBase.installExistingVmCheck(PreCheckBase.java:185) ~[eam-server.jar:?]
 at com.vmware.eam.agent.crud.impl.InstallAgentPreCheck.check(InstallAgentPreCheck.java:65) ~[eam-server.jar:?]
 at com.vmware.eam.agent.crud.impl.PreCheckBase.execute(PreCheckBase.java:101) ~[eam-server.jar:?]
 at com.vmware.eam.agent.crud.impl.InstallAgentPreCheck.execute(InstallAgentPreCheck.java:24) ~[eam-server.jar:?]
 at com.vmware.eam.job.InstallAgentJob.runWorkflow(InstallAgentJob.java:150) ~[eam-server.jar:?]
 at com.vmware.eam.job.AgentWorkflowJob.call(AgentWorkflowJob.java:93) ~[eam-server.jar:?]
 at com.vmware.eam.job.AgentWorkflowJob.call(AgentWorkflowJob.java:48) ~[eam-server.jar:?]
 at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) [eam-server.jar:?]
 at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) [eam-server.jar:?]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_311]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_311]
 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_311]

Environment

VMware vSphere ESXi 7.0.3
VMware vSphere ESXi 7.0.2

Cause

• Due to vSAN cluster shutdown workflow, vSAN disables the cluster in EAM service. Ultimately this powers off all EAM Virtual Machines in that cluster and EAM stops listening to PropertyCollector changes for the hosts in the cluster.
• Then due to VCSA restart, EAM service is restarted and EAM syncs the host properties for the cluster i.e. Powered-on and in Maintenance Mode.
• Later hosts are taken out of Maintenance Mode, but EAM has no listeners for the hosts and this is missed by EAM service.
• The next step is to enable the cluster in EAM service so that EAM can attach new listeners for the hosts and power on the Virtual Machines.
• However, due to the pre-check implementation, EAM does not power-on the vCLS Virtual Machines due to a stale host state.
• Furthermore, in case of restart while the cluster is disabled (like in this case) EAM is not attaching its listeners to the hosts, so EAM is blocked indefinitely with stale values for the host state.

Resolution

The issue has been resolved in VMware vSphere 7.0 U3o

Workaround:

To workaround the issue, please follow the below mentioned steps:

1. Restart EAM service.
2. Execute service-control --restart eam in VC shell.