Cluster Agent VM is missing on cluster

Products

VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

In the vCenter events of the affected cluster, you will see the below event:

Cluster Agent VM is missing on cluster (vCLS),Error,com.vmware.vim.eam,com.vmware.vim.eam.issue.cluster.agent.vmNotDeployed

In the /var/log/vmware/eam/eam.log on the vCenter, you may see below entries where vCLS VM deployment is failing

YYYY-MM-DDTHH:MM:SS.194Z | INFO | cluster-agent-0 | OvfDownloader.java | 79 | [OvfDownloader->Download:file:/storage/lifecycle/vmware-hdcs/photon-ova-5.0-24083810.ovf:ce2b43002baf1ecd] Downloading OVF descriptor from URL: file:/storage/lifecycle/vmware-hdcs/photon-ova-5.0-24083810.ovf

YYYY-MM-DDTHH:MM:SS.218Z | ERROR | cluster-agent-0 | AuditedJob.java | 106 | JOB FAILED: [#629236025] DeployVmJob(ClusterAgent(ID: ))
com.vmware.eam.job.DeployVmJob$DeployVmJobFailure: Can't provision VM for ClusterAgent(ID: ) due to lack of suitable datastore.

In the eam.log, you will see below entries repeating every 30 seconds

YYYY-MM-DDTHH:MM:SS.218Z | ERROR | cluster-agent-0 | AuditedJob.java | 106 | JOB FAILED: [#629236025] DeployVmJob(ClusterAgent(ID: ))
com.vmware.eam.job.DeployVmJob$DeployVmJobFailure: Can't provision VM for ClusterAgent(ID: ) due to lack of suitable datastore.

Environment

VMware vCenter Server 8.0 U3

Cause

Starting vSphere 8.0U3, Embedded vCLS is introduced. If any of ESXi host in the cluster is upgraded to 8.0U3, the cluster vCLS mode will change from External vCLS to Embedded vCLS. Embedded vCLS VM's are container based and run entirely in the ESXi memory
This issue occurs when the EAM service is struck in a retry loop trying to deploy External vCLS VM's on the Embedded vCLS cluster

Resolution

To resolve this issue:

Put the affected cluster in Retreat mode by referring to Disable vCLS on a Cluster via Retreat Mode
Wait for a few mins and revert the change by changing the vCLS mode to "System Managed"

Note: vSphere HA and DRS functionalities will be impacted while the cluster is in Retreat mode.

Additional Information

Refer Embedded vCLS for more details on Embedded vCLS

Certain Workflows on ESXi 8.0 U3 may cause DRS Downtime in Mixed-Version Clusters