Cluster Agent VM is missing on cluster (vCLS)

Products

• VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

• In the vCenter Server events of the affected cluster, you will see the below event:

Cluster Agent VM is missing on cluster <Cluster Name> (vCLS),Error,com.vmware.vim.eam,com.vmware.vim.eam.issue.cluster.agent.vmNotDeployed

• In the /var/log/vmware/eam/eam.log on the vCenter Server, you may see below entries where vCLS VM deployment is failing
  
  YYYY-MM-DDTHH:MM:SS.191Z |  INFO | cluster-agent-0 | AuditedJob.java | 106 | JOB STARTED: [#629236025] DeployVmJob(ClusterAgent(ID:))

YYYY-MM-DDTHH:MM:SS.194Z |  INFO | cluster-agent-0 | OvfDownloader.java | 79 | [OvfDownloader->Download:file:/storage/lifecycle/vmware-hdcs/photon-ova-5.0-24083810.ovf:ce2b43002baf1ecd] Downloading OVF descriptor from URL: file:/storage/lifecycle/vmware-hdcs/photon-ova-5.0-24083810.ovf

YYYY-MM-DDTHH:MM:SS.218Z | ERROR | cluster-agent-0 | AuditedJob.java | 106 | JOB FAILED: [#629236025] DeployVmJob(ClusterAgent(ID: ))
com.vmware.eam.job.DeployVmJob$DeployVmJobFailure: Can't provision VM for ClusterAgent(ID: ) due to lack of suitable datastore.

• In the eam.log, you will see below entries repeating every 30 seconds
  
  YYYY-MM-DDTHH:MM:SS.218Z | ERROR | cluster-agent-0 | AuditedJob.java | 106 | JOB FAILED: [#629236025] DeployVmJob(ClusterAgent(ID: ))
com.vmware.eam.job.DeployVmJob$DeployVmJobFailure: Can't provision VM for ClusterAgent(ID: ) due to lack of suitable datastore.

Environment

VMware vCenter Server 8.0 U3

Cause

• Starting vSphere 8.0U3, Embedded vCLS is introduced. If any of ESXi host in the cluster is upgraded to 8.0U3, the cluster vCLS mode will change from External vCLS to Embedded vCLS. Embedded vCLS VM's are container based and run entirely in the ESXi memory
• This issue occurs when the EAM service is struck in a retry loop trying to deploy External vCLS VM's on the Embedded vCLS cluster

Resolution

To resolve this issue:

• Put the affected cluster in Retreat mode by referring to Disable vCLS on a Cluster via Retreat Mode
• Wait for a few mins and revert the change by changing the vCLS mode to "System Managed"

Note: vSphere HA and DRS functionalities will be impacted while the cluster is in Retreat mode.

Additional Information

• Refer Embedded vCLS for more details on Embedded vCLS
• Certain Workflows on ESXi 8.0 U3 may cause DRS Downtime in Mixed-Version Clusters
  
  
• クラスタ エージェント仮想マシンがありません (vCLS)