Shutting down and powering on a vSAN Cluster when vCenter Server is running on top of vSAN
Article ID: 327033
Updated On:
Products
VMware vSAN
Issue/Introduction
This article provides instructions to safely shut down a vSAN cluster where the managing vCenter resides and graceful power-on.
Note: If vCenter Server is deployed on the vSAN datastore, it may not be available for troubleshooting if a problem occurs in the vSAN cluster. For related information, see the VMware vSAN Design Guide
Environment
VMware vSAN 7.0.x
VMware vSAN 6.6.x
VMware vSAN 6.7.x
VMware vSAN 6.2.x
VMware vSAN 6.1.x
VMware vSAN 6.5.x
VMware vSAN 6.0.x
VMware vSAN 6.6.x
VMware vSAN 6.7.x
VMware vSAN 6.2.x
VMware vSAN 6.1.x
VMware vSAN 6.5.x
VMware vSAN 6.0.x
Resolution
Shutting down the vSAN cluster
For 7.0 U3 shutdown process see Shutting Down and Restarting the vSAN Cluster.
Considerations before you begin:
- Use one of the following methods to confirm vSAN objects are healthy:
- Run the below command from RVC via vCenter once navigated to the cluster level to show a report of all vSAN objects in the cluster:
vsan.obj_status_report -t
- Run the below command from RVC via vCenter once navigated to the cluster level to show a report of all vSAN objects in the cluster:
Note: RVC will become obsolete soon
-
- Run the vSAN health script by running this command and pipe it to less then search for object:
python /usr/lib/vmware/vsan/bin/vsan-health-status.pyc |less - If running vSAN 6.6 or higher run "esxcli vsan debug object health summary get"
- Run the vSAN health script by running this command and pipe it to less then search for object:
- From vCenter go to vSAN Health "Object health" check, ensure there are no inaccessible objects.
To shut down the cluster:
1)Shut down all Virtual Machines running on the VSAN Cluster except for the vCenter Server VM, which will be the last VM you shut down.
2)To help simplify the startup process, migrate the vCenter Server VM to the first ESXi host so you can easily find the VM when powering back on your VSAN Cluster.
3)Ensure that there are no vSAN Components being resynced before shutting down the vCenter server. To check for resync do one of the following:
From RVC navigate to the cluster level then run vsan.resync_dashboard .
Note: RVC will become obsolete soon
From ESXI 6.5 or higher run esxcli vsan debug resync summary get
For ESXi below 6.5 run the below script
cd /tmp/; echo "" > ./resyncStats.txt ;cmmds-tool find -t DOM_OBJECT -f json |grep uuid |awk -F \" '{print $4}' |while read i;do pendingResync=$(cmmds-tool find -t DOM_OBJECT -f json -u $i|grep -o "\"bytesToSync\": [0-9]*,"|awk -F " |," '{sum+=$2} END{print sum / 1024 / 1024 / 1024;}');if [ ${#pendingResync} -ne 1 ]; then echo "$i: $pendingResync GiB"; fi;done |tee -a ./resyncStats.txt;total=$(cat resyncStats.txt |awk '{sum+=$2} END{print sum}');echo "Total: $total GiB" |tee -a ./resyncStats.txt;total=$(cat ./resyncStats.txt |grep Total); totalObj=$(cat ./resyncStats.txt|grep -vE " 0 GiB|Total"|wc -l);echo "`date +%Y-%m-%dT%H:%M:%SZ` $total ($totalObj objects)" >> ./totalHistory.txt; cat totalHistory.txt ; df -m | grep vsan
4) Shut down the vCenter Server VM which will now make the vSphere Web Client unavailable.
5) Place all nodes into maintenance mode by one of the following methods:
a) Direct connect to each node via the Web Client >Right click the node >Enter Maintenance Mode with No Action
b) From CLI of each node run esxcli system maintenanceMode set -e true -m noAction
6) Power down all nodes once in maintenance mode
2)To help simplify the startup process, migrate the vCenter Server VM to the first ESXi host so you can easily find the VM when powering back on your VSAN Cluster.
3)Ensure that there are no vSAN Components being resynced before shutting down the vCenter server. To check for resync do one of the following:
From RVC navigate to the cluster level then run vsan.resync_dashboard .
Note: RVC will become obsolete soon
From ESXI 6.5 or higher run esxcli vsan debug resync summary get
For ESXi below 6.5 run the below script
cd /tmp/; echo "" > ./resyncStats.txt ;cmmds-tool find -t DOM_OBJECT -f json |grep uuid |awk -F \" '{print $4}' |while read i;do pendingResync=$(cmmds-tool find -t DOM_OBJECT -f json -u $i|grep -o "\"bytesToSync\": [0-9]*,"|awk -F " |," '{sum+=$2} END{print sum / 1024 / 1024 / 1024;}');if [ ${#pendingResync} -ne 1 ]; then echo "$i: $pendingResync GiB"; fi;done |tee -a ./resyncStats.txt;total=$(cat resyncStats.txt |awk '{sum+=$2} END{print sum}');echo "Total: $total GiB" |tee -a ./resyncStats.txt;total=$(cat ./resyncStats.txt |grep Total); totalObj=$(cat ./resyncStats.txt|grep -vE " 0 GiB|Total"|wc -l);echo "`date +%Y-%m-%dT%H:%M:%SZ` $total ($totalObj objects)" >> ./totalHistory.txt; cat totalHistory.txt ; df -m | grep vsan
4) Shut down the vCenter Server VM which will now make the vSphere Web Client unavailable.
5) Place all nodes into maintenance mode by one of the following methods:
a) Direct connect to each node via the Web Client >Right click the node >Enter Maintenance Mode with No Action
b) From CLI of each node run esxcli system maintenanceMode set -e true -m noAction
6) Power down all nodes once in maintenance mode
Powering on the vSAN cluster
To power on the vSAN cluster:
1)Boot up all ESXi nodes that is part of the cluster either through remote console session or physically.
2)Connect to each ESXi host and exit maintenance mode using one of these options:
a)Using the vSphere Host Client
b)From the shell using this command: esxcli system maintenanceMode set -e false
3)If you followed the recommendation from step 2 of the shutdown process login into node 1 or the node you notated vCenter resides via the Web Client and power on your vCenter Server
4)Connect to your vCenter Server using the vSphere Web Client.
Note: It may take a few minutes for vCenter Server to become available again.
5)Do a quick health check on the vSAN cluster. Check for network partitions and resyncing components. For more information, see:
The Monitoring Virtual SAN section of the Administering VMware Virtual SAN guide.
The Check vSAN Health section of the Administering VMware Virtual SAN guide.
6)Power on the remaining virtual machines in the vSAN cluster.
2)Connect to each ESXi host and exit maintenance mode using one of these options:
a)Using the vSphere Host Client
b)From the shell using this command: esxcli system maintenanceMode set -e false
3)If you followed the recommendation from step 2 of the shutdown process login into node 1 or the node you notated vCenter resides via the Web Client and power on your vCenter Server
4)Connect to your vCenter Server using the vSphere Web Client.
Note: It may take a few minutes for vCenter Server to become available again.
5)Do a quick health check on the vSAN cluster. Check for network partitions and resyncing components. For more information, see:
The Monitoring Virtual SAN section of the Administering VMware Virtual SAN guide.
The Check vSAN Health section of the Administering VMware Virtual SAN guide.
6)Power on the remaining virtual machines in the vSAN cluster.
Additional Information
Impact/Risks:
Advisory: Please also review vSAN -- Maintenance -- Simultaneous Host Reboots / Cluster Full Shutdown -- Risk for Data Unavailability before shutting down the entire cluster.
Advisory: Please also review vSAN -- Maintenance -- Simultaneous Host Reboots / Cluster Full Shutdown -- Risk for Data Unavailability before shutting down the entire cluster.
Feedback
Yes
No
Powered by
