Error: vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster services caused by the unavailability of vSphere Cluster Service VMs.

Products

• VMware vCenter Server

Issue/Introduction

• Powered off vCLS VMs fails to delete even after enabling retreat mode on the affected cluster.
• In vCenter Server you can find the below log snippets in /var/log/vmware/wcp/wcpsvc.log 

  YYYY-MM-DDTHH:MM:SS.347Z error wcp [eamagency/retreat.go:49] [opID=vCLS] Unable to destroy agency (Agency:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx): ServerFaultCode: "com.vmware.eam.DisabledOperationException: Invocation of disabled API Agency.DestroyAgency"

Environment

vCenter 7.x

Resolution

• Remove the Retreat Mode configurations from the vpxd.cfg file using the following command.

# sed '/<vcls>/,/<\/vcls>/d' -i /etc/vmware-vpx/vpxd.cfg

Refer to Disable vCLS on a Cluster via Retreat Mode article For Versions Prior to vSphere 7.0 U3o and 8.0 U2, Using the vSphere Client for more information.

• Re-enable the API by leveraging the EAM MOB link here:

https://<vCenter_FQDN_or_IP_address>/eam/mob?vmodl=1

Under Methods section look for com.vmware.vim.binding.vmodl.ManagedObjectReference and click enable against it.

In the new window, modify the contents to include the "MOID" of the target cluster:

By default: <cluster type="ClusterComputeResource" serverGuid="Server GUID">MOID</cluster> 
Our modification: <cluster type="ClusterComputeResource" serverGuid="Server GUID">domain-cx</cluster> 

Click Invoke Method.

The powered off vCLS VMs were deleted automatically after the above change.

• Disable Retreat mode via vCenter GUI.
  Cluster > Configure > General > Edit vCLS Mode > System Managed.

New vCLS VMs were deployed and powered on successfully.

Additional Information