vSAN Shared Witness becomes isolated from all 2-node vSAN clusters when using reboot_helper.py shutdown prepare function.

Products

• VMware vSAN

Issue/Introduction

ESXi by default contains a built-in tool (reboot_helper.py - https://knowledge.broadcom.com/external/article/322144) that can be used for orchestrating simultaneous shutdown/reboot of all hosts in a vSAN cluster in edge-case where vSphere UI-based vSAN Shutdown Cluster Wizard (Shut Down the vSAN Cluster Using the Shutdown Cluster Wizard) is unavailable or not preferred. 

Environment

vSAN 8.x
vSAN 9.x

Cause

reboot_helper.py 'prepare' function actions all nodes in the cluster (regardless of type e.g. both data-nodes and Witness nodes) by triggering a synchronized vsan/witness vmk untag + store event and enabling parameter /VSAN/IgnoreClusterMemberListUpdates on all nodes simultaneously.

This intentionally results in a vSAN network isolation state for all nodes which this is triggered, including if they are a Shared Witness (e.g. connected to multiple vSAN clusters) where the 'prepare' function was only run on one data-node from one of the clusters this Witness is shared with.

Resolution

Do not use reboot_helper.py 'prepare' function on any vSAN cluster that configured to use a Shared Witness.

If a cluster using a Shared Witness needs to be shutdown then vSAN Shutdown Cluster Wizard should be used as this does not perform any actions on the Witness (regardless of whether it is Standalone or Shared) - if this is not a feasible option then a maintenance window for all clusters this Shared Witness is shared with needs to be arranged.

A cluster can be validated to be using a Standalone or Shared Witness via Cluster > Configure > Fault Domains > Select the configured Witness > Monitor > vSAN > Two Node Clusters (only a Shared Witness will show this UI pane):