vCLS VMs do not deploy due to the EAM Agency showing as "Disabled" following a vSAN Cluster Shutdown

Products

• VMware vCenter Server

Issue/Introduction

• vCLS VMs will not deploy in a cluster, even after enabling and disabling retreat mode
• Messages similar to the following are seen in vCenter - /var/log/vmware/vpxd/vpxd.log
  

  YYYY-MM-DDTHH:MM:SSZ info vpxd[#####] [Originator@6876 sub=Default opID=wcp-vCLS-####] [VpxLRO] -- ERROR lro-12345 -- VpxSettings -- vim.option.OptionManager.queryView: vim.fault.InvalidName:
  --> Result:
  --> (vim.fault.InvalidName) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = <unset>,
  -->    name = "config.vcls.clusters.",
  -->    entity = <unset>
  -->    msg = ""
  --> }
  --> Args:
  -->
  --> Arg name:
  --> "config.vcls.clusters."

• Recent logs from EAM show that the agency is disabled. vCenter - /var/log/vmware/eam/eam.log
  

  YYYY-MM-DDTHH:MM:SSZ |  INFO | vim-monitor | IssueHandlerBase.java | 116 |
  Updating issues:
  New issues:
   [
  eam.issue.AgencyDisabled {
     description = <unset>,
     time = YYYY-MM-DD HH:MM:SS,
     key = 0,
     agency = 'Agency:#######-#####-#####-#####-###########:null',
     solutionId =
  'VSPHERE.LOCAL\vpxd-extension-########-####-####-####-################',
     agencyName = 'vCLS',
     solutionName = ' ',
  }
  ]

Environment

• VMware vCenter Server
• VMware vSAN

Cause

This happens when the cluster's agencies are disabled in EAM. This usually follows the use of the vSAN Shutdown Wizard, where the Restart Cluster Wizard is not used to restart the cluster afterwards, which re-enables the EAM agencies as part of the workflow.

Resolution

In scenarios where EAM agencies are not functioning as expected following a vSAN shutdown, the following workaround should be applied.

For all future operations, it is strongly recommended to use the Restart Cluster Wizard after completing the vSAN Shutdown Wizard to ensure proper reinitialization of services.

Prerequisite:

Ensure that all virtual machines (VMs) running on the vSAN cluster are accessible and operating normally before proceeding.

Workaround Steps

1. Retrieve the Cluster MOID
   1. Open the vSphere Client and navigate to the vSphere cluster.
   2. Check the browser's address bar, where the cluster's MOID (Managed Object ID) will be shown in format of domain-c####. (e.g. domain-c10; do not copy anything after the ":")
2. Retrieve the Server GUID (especially important when using vCenter Enhanced Linked Mode)
   1. Open the vSphere Client and navigate to the relevant vCenter of affected cluster.
   2. Check the URL in the browser's address bar. The "Server GUID" will be shown in format of Folder:group-d1:########-####-####-####-############. The needed GUID is the 32 hexadecimal-digits string. (Do not copy "Folder:group-d1:" as they are not part of the GUID)
3.  To access the EAM Managed Object Browser (MOB), open a browser and go to:

   https://<VCENTER_IP>/eam/mob?vmodl=1

4. To enable EAM Agencies for the Cluster, click on the link under "Methods" called "enable". A pop-up window will appear.
5. In the value field, input the following XML, replacing "MOID" with the domain-c#### value and "Server GUID" with the actual server GUID:

   <cluster type="ClusterComputeResource" serverGuid="Server GUID">MOID</cluster>

6. Click "Invoke Method".
7. Wait for vCLS VMs to deploy.
8. Allow a few minutes for the cluster to automatically deploy the vSphere Cluster Services (vCLS) VMs.
9. A task named "Enable system VMs" will be triggered within the vCenter UI.

If the Task Fails

• Repeat Steps 1–5 above, and in Step 6 use the simplified XML (excluding serverGuid):

  <cluster type="ClusterComputeResource">MOID</cluster>

• Click Invoke Method.
• The "Enable system VMs" task should now complete successfully.

If vCLS VMs Are Still Not Deployed

Put the cluster into retreat mode, then disable retreat mode to force a refresh of the deployment process Reference KB Disable vCLS on a Cluster via Retreat Mode for guidance.

Additional Information

Despite the Supervisor VMs being healthy and all pods running, when the EAM agents are disabled, the WCP service can throw errors such as:

API call to VMware ESX Agent Manager (eam) failed. Details 'ServerFaultCode: "com.vmware.eam.DisabledOperationException: Invocation of disabled API Agent.markAsAvailable"

After restarting the vSAN cluster and enabling the EAM Agencies for the cluster from within the EAM Managed Object Browser (MOB), you may need to restart the WCP service on vCenter.

vmon-cli -r wcp