vSAN Cluster Partition and Inaccessible VMs After ESXi Upgrade from 7.0 to 8.0 vSAN
search cancel

vSAN Cluster Partition and Inaccessible VMs After ESXi Upgrade from 7.0 to 8.0 vSAN

book

Article ID: 453752

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

During upgrades from vSphere 7.0 (SHA-1 certificates) to 8.0 (SHA-256 certificates), vSAN clusters may experience a split-brain network partition. This issue occurs when nodes run mixed ESXi versions, causing Cluster Monitoring, Membership, and Directory Service (CMMDS) TLS handshake failures. vLCM may also prevent further host upgrades if cluster partition errors are detected.

  • Impact: vSAN cluster partitions, inaccessible objects, and VM "invalid" states.
  • Context: Upgrades performed via vLCM or manual methods where nodes remain on mixed software versions during the upgrade phase.

 

Environment

  • Source Version: ESXi 7.0.x (All builds).
  • Target Version: ESXi 8.0.x (All builds).

Cause

In vSphere 8.0, SHA-1 is deprecated in favor of SHA-256 for host certificates. When a cluster contains a mix of ESXi 7.0 nodes (using SHA-1) and upgraded ESXi 8.0 nodes (using SHA-256), the CMMDS daemon cannot establish a TLS handshake between peers. The nodes reject communication due to the hashing algorithm mismatch, leading to isolation.

Resolution

To resolve this issue, standardize all hosts in the cluster to the target ESXi 8.0 build. If vLCM cannot proceed, perform the following steps to force synchronization:

  1. Prevent vCenter Overrides: Run the following command on all hosts to prevent the partitioned vCenter from overriding manual changes: esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListupdates
  2. Extract New Thumbprints: Log in to the upgraded nodes via SSH and extract the SHA-256 thumbprint: openssl x509 -in /etc/vmware/ssl/rui.crt -fingerprint -sha256 -noout
  3. Purge Stale Entries: On the isolated/partitioned nodes, remove stale unicast entries: esxcli vsan cluster unicastagent remove -a
  4. Inject Updated Entries: Add the updated nodes back to the unicast agent list using the new thumbprint: esxcli vsan cluster unicastagent add -a <IP_Address> -u <Node_UUID> -t node -U true -c <Thumbprint>
  5. Reboot for Daemon Sync: Place the host in Maintenance Mode (Ensure Accessibility) and perform a full host reboot. This flushes cached certificates from the vSAN daemons.
  6. Verify: Check cluster health: esxcli vsan cluster get
  7. Revert Configuration: Once the cluster is unified and vCenter is accessible, revert the advanced configuration: esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates

Additional Information

  • If this issue persists after aligning versions, ensure all nodes have appropriate time synchronization (NTP) and that no stale firewall rules are blocking port 12321.
  • If you require further assistance from a Support Engineer, see our Contact Broadcom Support article.
Powered by Wolken Software