vCenter Server does not Automatically Provision VCLs Virtual Machines(VMs)

Products

• VMware vCenter Server

Issue/Introduction

Assist customers to solve this issue.

Symptoms:

The vCenter  Server does not automatically deploy vCLs after attempting retreat mode due to an agency in yellow status.

wcp.log remain in the deletion and destroying agent loop. There are no entries to create an agency.

2023-06-24T15:30:13.28Z debug wcp [dataprovider/cluster_agencies.go:96] [opID=vCLS] Successfully deleted from the database record (domain-c24137)
2023-06-24T15:30:13.28Z info wcp [eamagency/retreat.go:58] [opID=vCLS] Successfully destroyed agency, retreat operation completed

 

There  is no relevant  information in eam.log

In the eam_api-x.log you may see an error similar to the following:   

2023-06-20T14:29:13.130Z |  INFO | vlsi | LocalizationFilter.java | 108 | API COMPLETE: Facade for ClusterAgent(ID: 'Agent:3828c184-2c7a-4e15-9e2f-6059e3eb17a1:null').queryRuntime[opId=668781274, sessionId=2A280743]. Result:
eam.Agent.RuntimeInfo {
   issue = <unset>,
   goalState = 'enabled',
   entity = 'Agent:3328c105-2c3a-4e15-9e2f-6059e3eb17a1:null',
   status = 'yellow',
   vmPowerState = poweredOff,
   receivingHeartBeat = false,
   vmIp = <unset>,
   esxAgentResourcePool = <unset>,
   esxAgentFolder = <unset>,
   installedBulletin = <unset>,
   vmHook = <unset>,
   agency = 'Agency:10458969-7b9e-4ac9-b14e-45f6e2uyt5nvvc:null',
   vm = <unset>,
   installedVibs = <unset>,
   vmName = '',
   host = <unset>,

 

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x
VMware vCenter Server 7.0.0

Cause

There are EAM agencies in yellow status.

Resolution

1. Check if there are any agencies in yellow color.
2.  Restarting EAM usually fixes the problem.

service-control --stop vmware-eam && service-control --start vmware-eam

3. After restarting the services follow the steps 1 through 8 from https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-F98C3C93-875D-4570-852B-37A38878CE0F.html  or https://kb.vmware.com/s/article/91890?lang=en_US   to enable "Retreat Mode".
4. Wait for 1 to 2 minutes to make sure the VCLs VM's were removed from the cluster.
5. Disable retreat mode by changing the config.vcls.clusters.domain-c<number>.enabled  value to "true".

Additional Information

https://kb.vmware.com/s/article/91890?lang=en_US

Impact/Risks:

High because the DRS does not properly work causing an imbalance of cluster resources.