vSAN Host Isolated During Upgrade to 8.0

Products

• VMware vSAN
• VMware vSAN 7.x
• VMware vSAN 8.x

Issue/Introduction

When upgrading vSAN hosts from 7.0 hosts to 8.0 a network partition will occur between the upgraded 8.0 hosts, and the 7.0 hosts pending upgrade.

This even will be reported in vCenter as an alert - 'vSAN network alarm 'vSAN cluster partition.'

Vmkernel.log may contain the message - 'WARNING: CMMDS: CMMDSPromoteFormatVersion:423: Failed to promote the node to a format version X beyond its software version Y '

Environment

• vSAN OSA
• vSAN ESA

Cause

This manifestation of the Cluster Partition is a result of the On-Disk Version being updated on newly upgraded hosts, while non-upgraded hosts remain on the lower version.  This is known and expected behavior during this upgrade process.

Resolution

Prior to upgrade review the action plan below to ensure, to avoid unnecessary outages:

• Power down all running virtual machine (if only one upgraded node remains) or migrate virtual machine workloads to the remaining 7.0 host(s).
• Place the isolated host in Maintenance Mode (No Action)
• Place the upgrade package on a local /shared datastore

To upgrade the hosts, run the following commands from the ESX CLI:

1. esxcli software sources profile list -d '/vmfs/volumes/<Datastore_Name>/<Upgrade_Package_Name>.zip;

• • This Command provides the profile, required to the update command.

2. esxcli software profile update -p <Profile_Name> -d '/vmfs/volumes/<Datastore_Name>/<Upgrade_Package_Name>.zip' --no-hardware-warning --dry-run

Additional Information

• Failure to Promote CMMDS Version
  
  
• vSAN Node Down