vCLS VMs show as unknown after moving all ESXi hosts out of and back into a cluster

Products

• VMware vCenter Server

Issue/Introduction

vSphere Cluster Service (vCLS) VMs may enter an "Unknown" state with power operations greyed out.

Environment

7.0 U3

Cause

The vSphere Cluster Service loses the ability to track the state of the service VMs when the underlying host resources are completely removed. This breaks the link between the ESX Agent Manager (EAM) and the vCLS deployment engine.

Resolution

To resolve the issue follow the below steps.

vSphere 7.0 U3o/8.0 U2 and Later:

1. Log into vCenter's HTML5 client.
   
   
2. In Hosts and Clusters inventory, select a cluster.
   
   
3. Click on the Configure tab.
   
   
4. Under vSphere Cluster Services, select General.
   
   
5. In the top right, click on EDIT VCLS MODE.
   
   
   
   
   
6. In the Edit vCLS Mode pop up window, click on the second radio option "Retreat Mode."
   
   
   
   
7. Click OK.
   
   
8. Wait for 5–10 minutes. You should be able to see the problematic vCLS VMs being deleted in the recent tasks or under the Monitor > Tasks section.
   
   
9. Once all existing VMs have been removed, please change the vCLS mode back to "System managed."
   
   
   
   
10. Click OK.

Additional Information

Putting a Cluster in Retreat Mode

 

Note: Starting in vSphere 7.0 U3o and 8.0 U2, entering Retreat Mode is now available as a Cluster setting within the vCenter Server UI.