Virtual Machine power on operation fails with error "All required agent virtual machines are not currently deployed on host 'hostname'. Operation failed! error.mutationOperationFailed.unknownReason" due to stale eam agency

Products

• VMware vCenter Server

Issue/Introduction

When attempting to delete a stale vSphere Cluster Services (vCLS) agency using vSphere ESX Agent Manager (EAM), the following error is observed:

Operation failed! - error.mutationOperationFailed.unknownReason

Environment

• VMware vCenter Server 7.x

• VMware vCenter Server 8.x

Cause

This issue occurs when the ESX Agent Manager workflow fails to restart due to an existing stale cluster entry in the following configuration file:

/etc/vmware-vsan-health/config.conf

Example entry:

[PowerSystem] ha_for_domain-cxxxxxx = True

When this entry is present, the EAM cluster enablement step may be skipped if the workflow to enable HA fails. As a result, attempts to remove the stale vCLS agency do not succeed.

Resolution

Important: Before proceeding, take a snapshot of the vCenter Server appliance to ensure you can safely revert if needed.

Follow the steps below to manually re-enable the EAM cluster and remove the stale vCLS agency:

1. Manually Enable the EAM Cluster

a. Open the following URL in a web browser, replacing <vCenter_IP> with the IP address or FQDN of your vCenter Server:

https://<vCenter_IP>/eam/mob/?moid=EsxAgentManager&method=EsxAgentManager_Enable_Task&vmodl=1

b. In the value field, enter the following XML, replacing the placeholder values accordingly:

<cluster type="ClusterComputeResource" serverGuid="<vCenter Server GUID>">domain-cxxxxxx</cluster>

• Replace <vCenter Server GUID> with your vCenter server’s GUID.
  You can retrieve it by running the following command from the vCenter CLI:

grep instanceUuid /etc/vmware-vpx/instance.cfg

• Replace domain-cxxxxxx with the MOID of the affected cluster.
  To find the MOID:

  • Open the vSphere Client.

  • Click on the cluster in the inventory.

  • The URL will contain the cluster’s MOID. For example:

  https://<vCenter_IP>/ui/app/cluster;nav=h/urn:vmomi:ClusterComputeResource:domain-CXXXX:XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX/updates/

c. Click Invoke Method to execute the task and re-enable the EAM cluster.

2. Delete the Stale vCLS Agency

After successfully enabling the EAM cluster, attempt to delete the stale vCLS agency via the vSphere Client or using the appropriate API call.

Additional Information

Additional Information

• The Managed Object Browser (MOB) is a low-level interface and should be used with caution.

• If you're unsure of the MOID or Server GUID, you can retrieve them via the vCenter MOB or using PowerCLI.

• This issue can also affect other EAM-managed solutions if not resolved.
  
  

Impact/Risk

If not addressed, stale vCLS agencies may prevent proper cluster maintenance operations, such as enabling or disabling HA or modifying DRS settings.