Adding a host with newer version cause some VMs to go inaccessible

Products

• VMware vSAN

Issue/Introduction

• When you bring an ESXi host which is using a newer version compared to other hosts in the cluster, it can cause a network partition in the vSan cluster
•  Some VMs will go in inaccessible mode
• The disk group created on the newly added host will be a newer version
• Running  /usr/lib/vmware/vsan/bin/clom-tool stats | grep "minNodeMajorVersion" will give you a different version
  e.g: An ESXi with 8.0 u2 will show a minNodeMajorVersion of version 19 vs an ESXi 8.0 u3 which will show a version 20
• Running esxcli vsan clsuter get will show some hosts part of one network partition and the other hosts part of another 
• Hosts in the vSAN cluster have different versions

Environment

7.x

8.x

Cause

The change in minNodeMajorVersion between hosts of the same cluster will cause the vsan Cluster to get network partitioned

Resolution

Roll back the ESXi hosts that are on newer version. 

1. Validate if there is actually a lower build version available by checking the contents of /altbootbank/boot.cfg
2. Reboot the host
3. Press Shift+R repeatedly once the "system is preparing to boot" message appears
4. Once the ESXi host is rolled back and booted again the network partition will be gone