vCLS VMs do not deploy due to the EAM Agency showing as "Disabled" following a VSAN Cluster Shutdown

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCLS VMs will not deploy in a cluster, even after enabling and disabling retreat mode
• Messages similar to the following are seen in /var/log/vmware/vpxd/vpxd.log

2022-11-12T03:24:22.290Z info vpxd[62921] [Originator@6876 sub=Default opID=wcp-vCLS-1234] [VpxLRO] -- ERROR lro-12345 -- VpxSettings -- vim.option.OptionManager.queryView: vim.fault.InvalidName:
--> Result:
--> (vim.fault.InvalidName) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    name = "config.vcls.clusters.",
-->    entity = <unset>
-->    msg = ""
--> }
--> Args:
-->
--> Arg name:
--> "config.vcls.clusters."

• Recent logs from EAM show that the agency is disabled. /var/log/vmware/eam/eam.log

2022-05-24T10:44:08.619Z |  INFO | vim-monitor | IssueHandlerBase.java | 116 |
Updating issues:
New issues:
 [
eam.issue.AgencyDisabled {
   description = <unset>,
   time = 2023-04-24 11:11:08,594,
   key = 0,
   agency = 'Agency:12345678-1234-1234-1234-1234566778899:null',
   solutionId =
'VSPHERE.LOCAL\vpxd-extension-1234abcd-abcd-1234-1234-abcdef123456',
   agencyName = 'vCLS',
   solutionName = ' ',
}
]

Environment

VMware vCenter Server 7.0.x

Cause

This happens when the cluster's agencies are disabled in EAM. This usually follows the use of the VSAN Shutdown Wizard, where the Restart Cluster Wizard is not used to restart the cluster afterward, which re-enables the EAM agencies as part of the workflow.

Resolution

This must be resolved with the workaround below. For future operations, please make sure that the Restart Cluster Wizard is used after the VSAN Shutdown Wizard.

Workaround:
Before starting, ensure the VMs on the vsan cluster are running without accessibility issues

To re-enable the EAM agencies on the cluster:

1. Obtain the clusters MOID by selecting it in the vSphere Client and using the address bar to find the domain-c#### identifier.
2. From a browser, access the EAM managed object browser (MOB)

https://<VCENTER_IP>/eam/mob?vmodl=1

1. Click on Enable and it will open a pop-up window.
2. In the value field "<cluster type="ClusterComputeResource" serverGuid="Server GUID">MOID</cluster>" replace MOID with the domain-c#### value you collected in step 1.
3. Click Invoke Method.
4. Give the cluster a few minutes after the enablement to deploy the vCLS VMs.

As soon as you click on invoke method, "Enable system VMs" task till be triggered on vCenter cluster.

If the task fails, you need to again follow the steps and remove the serverGUID part from step 2 and only add below line:

"<cluster type="ClusterComputeResource">MOID</cluster>" replace MOID with the domain-c#### value you collected in step 1.

 

Once invoked, Enable system VMs task will show as successful.  Even after waiting for 10 mins, if vCLS VMs are not deployed, you can put cluster in retreat mode and disable retreat mode to take effect.

 

Disable vCLS on a Cluster via Retreat Mode