Shutting down and powering on a vSAN Cluster when vCenter Server is running on top of vSAN

Products

• VMware vSAN

Issue/Introduction

Instructions to shutdown & power-on vSAN cluster when the managing vCenter resides on it. 

 

Note: If vCenter Server is deployed on the same vSAN datastore being shutdown, it may not be available for troubleshooting if a problem occurs in the vSAN cluster. For related information, see the VMware vSAN Design Guide

 

 

Impact/Risks:

Advisory: Please also review vSAN -- Maintenance -- Simultaneous Host Reboots / Cluster Full Shutdown -- Risk for Data Unavailability  before shutting down the entire cluster.

Environment

VMware vSAN 6.X
VMware vSAN 7.X
VMware vSAN 8.X

Resolution

Shutting down the vSAN cluster

Considerations before you begin:

• Use one of the following methods to confirm vSAN objects are healthy:

  • Run the below command from RVC via vCenter once navigated to the cluster level to show a report of all vSAN objects in the cluster:

    vsan.obj_status_report -t

    Note: RVC will become obsolete soon

• • Run the vSAN health script by running this command and pipe it to less then search for object:

    python /usr/lib/vmware/vsan/bin/vsan-health-status.pyc |less

  • If running vSAN 6.6 or higher run "esxcli vsan debug object health summary get"

• From vCenter go to vSAN Health "Object health" check, ensure there are no inaccessible objects.

To shut down the vSAN cluster:

1. Shut down all Virtual Machines running on the VSAN Cluster except for the vCenter Server VM, which will be the last VM you shut down.

2. To help simplify the startup process, migrate the vCenter Server VM to the first ESXi host so you can easily find the VM when powering back on your VSAN Cluster.

3. Ensure that there are no vSAN Components being resynced before shutting down the vCenter server. To check for resync do one of the following:

   From RVC navigate to the cluster level then run vsan.resync_dashboard .

   From ESXI 6.5 or higher run esxcli vsan debug resync summary get

   For ESXi below 6.5 run the below script

   cd /tmp/; echo "" > ./resyncStats.txt ;cmmds-tool find -t DOM_OBJECT -f json |grep uuid |awk -F \" '{print $4}' |while read i;do pendingResync=$(cmmds-tool find -t DOM_OBJECT -f json -u $i|grep -o "\"bytesToSync\": [0-9]*,"|awk -F " |," '{sum+=$2} END{print sum / 1024 / 1024 / 1024;}');if [ ${#pendingResync} -ne 1 ]; then echo "$i: $pendingResync GiB"; fi;done |tee -a ./resyncStats.txt;total=$(cat resyncStats.txt |awk '{sum+=$2} END{print sum}');echo "Total: $total GiB" |tee -a ./resyncStats.txt;total=$(cat ./resyncStats.txt |grep Total); totalObj=$(cat ./resyncStats.txt|grep -vE " 0 GiB|Total"|wc -l);echo "`date +%Y-%m-%dT%H:%M:%SZ` $total ($totalObj objects)" >> ./totalHistory.txt; cat totalHistory.txt ; df -m | grep vsan

4. Shut down the vCenter Server VM which will now make the vSphere Web Client unavailable.

5. Place all nodes into maintenance mode by one of the following methods:

   a) Direct connect to each node via the Web Client >Right click the node >Enter Maintenance Mode with No Action

   b) From CLI of each node run esxcli system maintenanceMode set -e true -m noAction

6. Power down all nodes once in maintenance mode

    

 

Powering ON the vSAN cluster

To power ON the vSAN cluster:

1. Boot up all ESXi nodes that is part of the cluster either through remote console session or physically.

2. Connect to each ESXi host and exit maintenance mode using one of these options:

   a)Using the vSphere Host Client

   b)From the shell using this command: esxcli system maintenanceMode set -e false

3. If you followed the recommendation from step 2 of the shutdown process login into node 1 or the node you notated vCenter resides via the Web Client and power on your vCenter Server.

4. Connect to your vCenter Server using the vSphere Web Client. Note: It may take a few minutes for vCenter Server to become available again.

5. Do a quick health check on the vSAN cluster. Check for network partitions and resyncing components.

6. Power ON the remaining virtual machines in the vSAN cluster.