Active vCenter management connection is lost after powering off Passive and Witness nodes

Products

• VMware vCenter Server

Issue/Introduction

• When attempting to perform maintenance tasks or a vCenter High Availability (VCHA) destroy operation, the following behavior is observed, upon powering off the Passive and Witness nodes, the vSphere Web Client connection to the Active node immediately disconnects

• The Management IP address (NIC 0 / eth0) on the Active node fails to link up or drops completely. General management connectivity to the Active vCenter Server is lost until the peer nodes are brought back online.

Environment

• VMware vCenter Server

Cause

• This behavior is an intentional design of the vCenter HA mechanism to prevent a Split-Brain condition.
• In a VCHA configuration, the cluster requires a majority of nodes (two out of three) to be online to maintain "Quorum." If both the Passive and Witness nodes are powered off, the Active node lose its ability to communicate with its peers. Without a majority, the Active node cannot verify that it is the legitimate primary node for the cluster. To protect the environment from data corruption or conflicting management actions, the vCenter Server software automatically disables the management network interface (NIC 0) until Quorum is restored.

Resolution

To preserve management connectivity while peer nodes are offline, the vCenter HA cluster must be placed into Maintenance Mode before any nodes are powered down. This configuration allows the Active node to remain operational even without a cluster majority.

     Enter Maintenance Mode:

1. Navigate to the vCenter Server in the vSphere Client inventory.
2. Select Configure > vCenter HA
3. Click Edit and select Maintenance Mode
   Note:  Automatic failover is suspended, but the management interface (NIC 0) on the Active node will stay online regardless of the status of the peer nodes going offline.
4. Then power off Passive and Witness node
5. The Active node will remain accessible via web client, take Putty/SSH access on Active Node as "root" user and run destroy VCHA command :

   vcha-destroy -f

6. Once the VCHA is destroyed, safely delete the Passive and Witness Node

Additional Information

Additional Reference: https://knowledge.broadcom.com/external/article/427556

https://knowledge.broadcom.com/external/article/427313/how-to-remove-a-vcenter-high-availabilit.html