vSAN Network Partition after downgrading ESXi host to 8.0 Update 2 from 8.0 Update 3

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms

After downgrading an ESXi host from version 8.0 Update 3 to 8.0 Update 2 within a vSAN cluster, the following symptoms are observed:

• The downgraded host reports a "Network Partition" in vSAN Skyline Health.
• The vSAN cluster appears heterogeneous, with some hosts on ESXi 8.0 U3 and the downgraded host on ESXi 8.0 U2.
• Virtual Machine (VM) replication or migrations to/from the downgraded host fail.
• vSAN Skyline Health may report "Disk format version warning" or "Reduced availability" for all objects.
• In vmkernel.log, you may see warnings regarding CMMDS promotion failures or version mismatches.

Environment

VMware vSAN 8.X

Cause

This issue occurs because vSAN on-disk format version 20 was introduced with ESXi 8.0 Update 3.

• On-disk format version 20 is not backward compatible with ESXi 8.0 Update 2.
• When the ESXi binary is downgraded while the disk groups retain the newer format (v20), the host can no longer properly communicate or maintain membership with the remaining 8.0 U3 cluster members.
• The incompatibility prevents the host from participating in the distributed object replication, resulting in a network partition.

Resolution

To resolve the network partition and restore replication functionality, the compute layer must be aligned with the on-disk metadata version.

1. Identify Affected Host: Verify the ESXi version of all hosts in the cluster. Confirm that the partitioned host is running ESXi 8.0 Update 2 while the disk format is version 20.
2. Upgrade ESXi Host: Re-install or upgrade the affected host to ESXi 8.0 Update 3 (Build 24585383) or higher to match the rest of the cluster.
3. Verify Cluster Health:
   • Navigate to vSAN Skyline Health in the vSphere Client.
   • Select Retest to refresh the cluster status.
   • Confirm the "Network Partition" alert is cleared.
4. Monitor Object Resynchronization: Ensure that all vSAN objects transition to a "Healthy" state and that any pending data resynchronization completes successfully.

Additional Information

Understanding vSAN on-disk format versions and compatibility.