vCenter High Availability failover due to primary node network isolation

Products

• VMware vCenter Server 8.0

Issue/Introduction

• A vCenter Server High Availability (VCHA) failover event is observed.
• In /var/log/vmware/vcha/vcha.log on the primary node:
  

  
[YYYY-MM-DDTHH:MM:SS] error vcha[#####] [Originator@6876 sub=Cluster opID=clusterManager.cpp:###-########] Timeout for slave @ <Secondary_Node_IP>
[YYYY-MM-DDTHH:MM:SS] verbose vcha[#####] [Originator@6876 sub=Cluster opID=clusterManager.cpp:###-########] Marking slave <Secondary_Node_IP> as unreachable

  [YYYY-MM-DDTHH:MM:SS] error vcha[#####] [Originator@6876 sub=Cluster opID=clusterManager.cpp:###-########] Timeout for slave @ <Witness_Node_IP>
[YYYY-MM-DDTHH:MM:SS] verbose vcha[#####] [Originator@6876 sub=Cluster opID=clusterManager.cpp:###-########] Marking slave <Witness_Node_IP> as unreachable

[YYYY-MM-DDTHH:MM:SS] info vcha[#####] [Originator@6876 sub=ClusterMgr opID=WorkQueue-########] Node is isolated



• In /var/log/vmware/vcha/vcha.log on the secondary node:

  
[YYYY-MM-DDTHH:MM:SS] info vcha[#####] [Originator@6876 sub=Election opID=clusterElection.cpp:###-########] Election State Change Slave => Startup : Lost master

  [YYYY-MM-DDTHH:MM:SS] verbose vcha[#####] [Originator@6876 sub=Cluster opID=clusterManager.cpp:###-########] Transitioned from election state Candidate to Master

Cause

The failover occurred because the primary node became network isolated from the secondary and witness nodes.

In a VCHA cluster, nodes exchange heartbeat messages over the VCHA network. If the primary node stops receiving heartbeats from the secondary and witness nodes, it considers the nodes unreachable.

When the primary node becomes isolated, the secondary node initiates an election process and takes over as the new primary node, triggering a VCHA failover.

Resolution

Ensure that the network used by the VCHA cluster is stable and meets the required network conditions.

VCHA requires a highly reliable network between the active, passive, and witness nodes, with latency below 10 ms. Refer to: vCenter HA Hardware and Software Requirements