Updating a 2-node stretched cluster running vSAN 7.03 with a shared witness host to ESXi 8.03 by replacing the witness caused a cluster partition.

Products

• VMware vSAN
• VMware vSAN 7.x
• VMware vSAN 8.x

Issue/Introduction

Attempted upgrade of a 2-node stretched cluster (vSAN 7.03) using a shared vSAN Witness host by replacing the witness with a new one running ODF version 20 (8 U3) resulted in cluster partitioning.

Environment

VMware vSAN 7.x (VMware vSAN 7.x)
VMware vSAN 7.x (VMware vSAN 8.x)
Shared vSAN Witness node

Cause

• The root cause was an ODF version mismatch:
• Hosts were running ODF version 15.
• The witness was running ODF version 20.
• This mismatch led to incompatible CMDS versions between nodes.
• Adding or recreating disk groups on hosts with higher ODF versions updated their CMMDS version, making them incompatible with lower-version nodes.

Resolution

1. If the issue has already occurred, proceed with updating the remaining hosts to restore cluster functionality.
2. Upgrade all ESXi hosts in the cluster to version 8.x before upgrading the witness.
3. Avoid adding/removing nodes or creating new disk groups on updated hosts during the upgrade.
4. Do not upgrade the ODF version until all hosts connected to the witness are running ESXi 8.x.
5. Upgrading all hosts to ESXi 8.x quickly resolves version incompatibility and restores cluster communication.

Additional Information

• When upgrading a stretch cluster using a shared witness, as a best practice, always upgrade the witness after all the ESXi hosts in the cluster.
• When upgrading a stretch cluster using a standalone witness with target builds of 7.0 or higher, always upgrade the witness host before the ESXi hosts.
• Mixed versions are allowed temporarily during upgrades, but avoid cluster modifications during this time.

For more information, also see 

Failure to promote CMMDS version, resulting in vSAN cluster becoming partitioned during upgrade

 VMware vSAN Upgrade Best Practices