After vCenter upgrade to 8.0 Update 3g , vCenter tasks are getting stuck or queued

Products

• VMware vCenter Server

Issue/Introduction

• vCenter tasks are stuck and recent tasks are queued on UI 

• /var/log/vmware/eam.log reports the below error 

YYYY-MM-DD| ERROR | cluster-agent-4 | AuditedJob.java | 106 | JOB FAILED: [###########] UpgradeClusterAgentJob(ClusterAgent(ID: ##########))
com.vmware.eam.job.DeployVmJob$DeployVmJobFailure: Can't provision VM for ClusterAgent(ID: ##########) due to lack of suitable datastore.
        at com.vmware.eam.job.DeployVmJob.doJob(DeployVmJob.java:377) ~[eam-server.jar:?]
        at com.vmware.eam.job.DeployVmJob.call(DeployVmJob.java:248) ~[eam-server.jar:?]
        at com.vmware.eam.job.DeployVmJob.call(DeployVmJob.java:128) ~[eam-server.jar:?]
        at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) [eam-server.jar:?]
        at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) ~[eam-server.jar:?]
        at com.vmware.eam.async.impl.SameThreadJ2SEExecutor.execute(SameThreadJ2SEExecutor.java:18) ~[eam-server.jar:?]
        at com.vmware.eam.async.impl.Executor.submit(Executor.java:40) ~[eam-server.jar:?]
        at com.vmware.eam.job.AgentWorkflowJob.execute(AgentWorkflowJob.java:166) ~[eam-server.jar:?]
        at com.vmware.eam.job.ClusterAgentWorkflowControl.execute(ClusterAgentWorkflowControl.java:35) ~[eam-server.jar:?]
        at com.vmware.eam.job.UpgradeAgentJobBase.deployNewAgentVM(UpgradeAgentJobBase.java:446) ~[eam-server.jar:?]
        at com.vmware.eam.job.UpgradeAgentJobBase.deployAgentVms(UpgradeAgentJobBase.java:417) ~[eam-server.jar:?]
        at com.vmware.eam.job.UpgradeAgentJobBase.lambda$deployNewAgentVm$1(UpgradeAgentJobBase.java:196) ~[eam-server.jar:?]
        at com.vmware.eam.agency.impl.ClusterVmDeploymentLock.lambda$withLock$1(ClusterVmDeploymentLock.java:42) ~[eam-server.jar:?]
        at com.vmware.eam.lock.RaceProtection.exec(RaceProtection.java:89) ~[eam-server.jar:?]
        at com.vmware.eam.lock.RaceProtection.exec(RaceProtection.java:59) ~[eam-server.jar:?]
        at com.vmware.eam.agency.impl.ClusterVmDeploymentLock.withLock(ClusterVmDeploymentLock.java:39) ~[eam-server.jar:?]
        at com.vmware.eam.job.UpgradeAgentJobBase.deployNewAgentVm(UpgradeAgentJobBase.java:179) ~[eam-server.jar:?]
        at com.vmware.eam.job.UpgradeAgentJob.runWorkflow(UpgradeAgentJob.java:139) ~[eam-server.jar:?]
        at com.vmware.eam.job.AgentWorkflowJob.call(AgentWorkflowJob.java:93) ~[eam-server.jar:?]
        at com.vmware.eam.job.AgentWorkflowJob.call(AgentWorkflowJob.java:48) ~[eam-server.jar:?]
        at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) [eam-server.jar:?]
        at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) [eam-server.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_452]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_452]

Environment

vCenter server 8.x 

ESXI server 7.x

ESXI server 8.x

Cause

• This issue occurs due to mixed version of Esxi in the cluster.
• From 8 update 3 , vCLS VM are embedded Embedded vSphere Cluster Services Overview

 

Resolution

Upgrade all the ESXI on the cluster to 8 update 3