Incorrect clusterAgent cluster membership information on ESXi

Products

• VMware vSphere ESX 8.x

Issue/Introduction

• Within a vSphere cluster, ESXi hosts report inconsistent clusterAgent membership information when the following command is executed:
  
  /usr/lib/vmware/clusterAgent/bin/clusterAdmin cluster status

The observed behavior includes:
  • Three ESXi hosts report a hosted state and the correct cluster ID.
  • Some ESXi hosts report a hosted state but display a different cluster ID.
  • The remaining ESXi hosts report a standalone state.
• Example output from /var/run/log/clusterAgent.log on hosts reporting an incorrect Cluster ID:
  
  [YYYY-MM-DDTHHH:MM:SS] No(5) clusterAgent[#######]: WARN grpc: addrConn.createTransport failed to connect to {<ESXi_IP>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <ESXi_IP>:2379: connect: connection refused". Reconnecting...

Environment

VMware vCenter Server 8.x

VMware vSphere ESXi 8.x

Cause

Each vSphere cluster maintains an embedded etcd cluster consisting of three ESXi hosts. These hosts act as the etcd members and therefore report a hosted state. All other ESXi hosts in the cluster are expected to report a standalone state.

This issue may occur when an ESXi host was previously a member of a different vSphere cluster and was later moved to a new cluster. In some cases, the host may retain stale clusterAgent or etcd membership information from the previous cluster. As a result, the host reports a hosted state associated with an outdated cluster ID.

Resolution

Verify the correct cluster ID of the vSphere cluster and force the affected host to leave the stale cluster membership information.

Verify the cluster ID

1.  Log in to the vSphere Client.
2. Navigate to the affected cluster.
3. Record the cluster domain ID from the browser URL. The Cluster ID is displayed in the format: domain-c<number>.
   For example：
   
   https://<vcenter-fqdn>/ui/app/cluster;nav=h/urn:vmomi:ClusterComputeResource:domain-c<number>:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/summary
   
   The Cluster ID in this example is: domain-c<number>

Force the affected host into standalone state

1. Connect to the affected ESXi host through SSH:
2. Force the host into standalone mode:
   
   /usr/lib/vmware/clusterAgent/bin/clusterAdmin cluster forceStandalone
   
   
3. Verify the host status again:
   
   /usr/lib/vmware/clusterAgent/bin/clusterAdmin cluster status
   
   The host should now report a standalone state.