vCenter services will not start on any node in VCHA configuration: Node is isolated

Products

• VMware vSphere ESXi

Issue/Introduction

In a VCHA configured environment the only services running are observability, vmware-vcha, vmware-vdtc, vmware-vmon. When attempting to start any other service they immediately fail. 

Looking at the vcha logs you can see where the nodes are unable to communicate with each other and they report node isolation.

info vcha[1311190] [Originator@6876 sub=ClusterMgr opID=WorkQueue-3933d9d3] Node is isolated

Environment

vCenter environments using VCHA

Cause

Upstream networking issues that prevent the nodes from communicating on their eth1 (VCHA) network will cause all the nodes to report as isolated. Any node reporting as isolated will not allow any of the VC services to start due to the potential for a split brain condition. 

Resolution

Either remediate the networking configuration issues so the VCHA can communicate on their network configured on eth1 or use the vcha-destroy command on the primary node to remove the VCHA configuration which will allow you to start the services there.