vCenter UI timeouts due to VCHA instability

Products

• VMware vCenter Server

Issue/Introduction

• vCenter UI timeouts or intermittent connectivity drops
• vCenter Server High Availability (VCHA) cluster experienced frequent role reversals between the Active and Passive nodes.
• vCenter HA nodes report as "Isolated" or "Lost Master."
  Logs in /var/log/vmware/vcha/vcha.log report:
  • info vcha[####] [Originator@#### sub=Agent] Lost master
  • info vcha[####] [Originator@#### sub=Agent] Slave timed out
• ESXi host logs (/var/log/vmkernel.log) show:
  • I/O latency increased from #### microseconds to #### microseconds

Environment

VMware vCenter Server 8.0.x

Cause

High storage I/O latency on the datastore hosting the vCenter Server Appliance prevents successful heartbeat communication between the VCHA nodes. When latency exceeds the 10ms (10,000 microseconds) Round Trip Time (RTT) threshold, the cluster assumes a node failure and triggers a failover.

Resolution

 To identify and validate the impacted datastore for the vCenter nodes during  high latency event, follow these steps:

Identify the Impacted Datastore

1. Run the following command to find the specific NAA ID reported for high I/O latency: 
   cat /var/log/vmkernel.log | grep "I/O latency increased"
2. Map the Datastore Name to the NAA ID (Device Name) using the localcli command: 
   localcli storage vmfs extent list
3. Verify the vCenter VM (Active, Passive or Witness) placement by
   logging into the ESXi Host Client, navigating to Virtual Machines, and checking the datastore path in the Summary tab.

Remediation

1. Remove the VCHA configuration via the vCenter CLI to prevent unpredictable failovers while the storage issue persists: 
   
   vcha-destroy -f
2. Contact your storage administrator to perform a comprehensive review of the SAN fabric and storage array to resolve the underlying I/O latency.
3. Once storage stability is confirmed were latency remains below 10ms, re-enable the VCHA configuration.