Recovering Supervisor Cluster when ESX host is disconnected from VCenter with a powered-on Control Plane Virtual Machine

Products

• VMware vCenter Server

Issue/Introduction

• Providing instructions to recover a Supervisor Cluster if an ESX host disconnects from VCenter with a powered-on     CP VM.
• Workload Control Plane doesn't support disconnecting a host while keeping the CP VM powered on.

Symptoms:
An ESX host disconnects from VCenter with a powered-on Control Plane Virtual Machine (CP VM).

Cause

An ESX host disconnects from VCenter with a powered-on CP VM.

Resolution

If an ESX host disconnects with a powered-on CP VM, on purpose or otherwise. Follow the below steps:

If the ESX host can be reconnected:
  1. Reconnect the ESX host.
  2. Put the host in the MM (Maintenance Mode).
  3. This will result in all Virtual Machines on the host being relocated to another host.
      No downtime should occur.
  4. Disconnect the host that is in maintenance mode.
  5. Safely remove the disconnected host.

If the ESX host can NOT be reconnected:
  1. Directly log into the ESX host and shut down the CP VM which  is running.
  2. This will result in the Supervisor Cluster only having 2 CP nodes, and one of them will take
      over as leader if the offline one was the leader.
  3. Remove the ESX host from the vCenter inventory.
  4. This will result in the EAM deploying a new Virtual Machine for the agency.

Please note in either of the above situations, it is of prime importance to replace the removed host as soon as possible to avoid an unrecoverable cluster or unexpected issues.