Rebuilding a vSAN cluster after reinstalling ESXi on all hosts without vCenter while preserving production data

Products

• VMware vSAN

Issue/Introduction

• vCenter resides on the vSAN datastore and can't be redeployed 
• ESXi has to be reinstalled on all hosts

Environment

VMware vSAN OSA (All Versions)
VMware vSAN ESA (All Versions)

Resolution

Note: Do not make any changes to the disks or disk groups (OSA) / storagepools (ESA), otherwise the data on the disks will be lost.

The below steps start after ESXi has already been reinstalled on all hosts.

Step 1 – Create the VMKernel interface to be used for vSAN
In order for vSAN to function, each host/node requires a VMKernel interface with the vSAN service enabled on it. This requires creating a temporary standard vSwitch and a Port Group to reform the vSAN cluster. If a Distributed vSwitch was previously used you can switch back over once vCenter is back online as Distributed vSwitch can only be configured/managed by vCenter. 

The following steps include steps for both Host UI or ESXi CLI depending on preference and availability.

To create a Standard vSwitch via Host UI see the below Tech Docs:

Add a Standard Virtual Switch in theVMware Host Client
Configuring the vSAN Network

To create a Standard vSwitch via ESXi CLI:

esxcli network vswitch standard add -v <vswitch_name>

Example:

esxcli network vswitch standard add -v vSwitch1

Once 'vSwitch1' is created, add the physical uplinks to it. to help identify which uplinks to use, run the following command:

esxcli network nic list

This should return details on all the physical network cards on the host for example:

Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:01:00.0 ntg3 Up 1000Mbps Full 44:**:**:**:**:98 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic1 0000:01:00.1 ntg3 Up 1000Mbps Full 44:**:**:**:**:99 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic2 0000:02:00.0 ntg3 Down 0Mbps Half 44:**:**:**:**:9a 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic3 0000:02:00.1 ntg3 Down 0Mbps Half 44:**:**:**:**:9b 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic4 0000:82:00.0 ixgbe Up 10000Mbps Full a0:**:**:**:**:cc 1500 Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
vmnic5 0000:82:00.1 ixgbe Up 10000Mbps Full a0:**:**:**:**:ce 1500?? Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
vmnic6 0000:04:00.0 ixgbe Up 10000Mbps Full a0:**:**:**:**:c4 1500 Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
vmnic7 0000:04:00.1 ixgbe Up 10000Mbps Full a0:**:**:**:**:c6 1500 Intel Corporation Ethernet Controller 10 Gigabit X540-AT2

In the following example, vmnic5 is added to 'vSwitch1'.

To add a vmnic to the standard switch created in the last step, run the following command:

esxcli network vswitch standard uplink add -v <vSwitch_name> -u vmnic<X>

Example:

esxcli network vswitch standard uplink add -v vSwitch1 -u vmnic5

Next, configure a portgroup for vSAN, In this example, the portgroup name is “vSAN”

esxcfg-vswitch -A <portgroup_name> <vSwitch_name>

Example:

esxcfg-vswitch -A vSAN vSwitch1

To create a vmkernel interface in the protgroup just created, run the following:

esxcfg-vmknic -a -i <vsan_kernel_IP> -n <subnet_mask> -p <portgroup_name>

Example:

esxcfg-vmknic -a -i 192.####.###.1 -n 255.255.255.0 -p vSAN

Validate the vmkernel Interface by running the following command:

esxcfg-vmknic -l

To enable the vSAN service on the new VMKernel interface, run the following command:

esxcli vsan network ip add -i vmk<X>

Example:

esxcli vsan network ip add -i vmk1

Repeat the above steps on the remaining hosts to be added to the cluster

Step 2 – Creating the cluster
Once all the VMKernel interfaces are configured on all the hosts, create a vSAN Cluster on any one host.

To do this we run the following command:

esxcli vsan cluster new

View and take note of the new vSAN Sub-Cluster UUID by running the following command:

esxcli vsan cluster get

Example output:

[root@se-emea-vsan01:~] esxcli vsan cluster get
Cluster Information
 Enabled: true
 Current Local Time: 2016-11-21T15:17:57Z
 Local Node UUID: 582*****-****-****-****-a***********4
 Local Node Type: NORMAL
 Local Node State: MASTER
 Local Node Health State: HEALTHY
 Sub-Cluster Master UUID: 582*****-0fd8-****-7460-a***********c
 Sub-Cluster Backup UUID: 582*****-cbfc-****-f794-a***********4
 Sub-Cluster UUID: 52bca225-0520-fd68-46c4-************
 Sub-Cluster Membership Entry Revision: 6
 Sub-Cluster Member Count: 1
 Sub-Cluster Member UUIDs: 582*****-cbfc-****-f794-a***********4
 Sub-Cluster Membership UUID: d2d*****-****-bbb9-9e1a-a***********c

Step 3 – Adding the other nodes to the cluster
From the remaining hosts run the following command adding them to the newly created cluster:

esxcli vsan cluster join -u <cluster_uuid>

Example (using the Sub-Cluster UUID from the sample output from Step 2):

esxcli vsan cluster join -u 52bca225-0520-fd68-46c4-************

You can verify that the nodes have successfully joined the cluster by running the same command we ran earlier noting that the Sub-Cluster Member Count has increased to the number of nodes participating in the cluster and it also shows the other sub cluster UUID Members:

[root@se-emea-vsan01:~] esxcli vsan cluster get
Cluster Information
 Enabled: true
 Current Local Time: 2016-11-21T15:17:57Z
 Local Node UUID: 582a29ea-cbfc-195e-f794-************
 Local Node Type: NORMAL
 Local Node State: MASTER
 Local Node Health State: HEALTHY
 Sub-Cluster Master UUID: 582*****-0fd8-****-7460-a***********c
 Sub-Cluster Backup UUID: 582*****-cbfc-****-f794-a***********4
 Sub-Cluster UUID: 52bca225-0520-fd68-46c4-************
 Sub-Cluster Membership Entry Revision: 6
 Sub-Cluster Member Count: 3
 Sub-Cluster Member UUIDs: 582a29ea-cbfc-195e-f794-************, 582a2bf8-4e36-abbf-5318-************, 582a2c3b-d104-b96d-d089-************
 Sub-Cluster Membership UUID: d2d*****-****-bbb9-9e1a-a***********c

Step 4 - Browse the vSAN datastore and get the old cluster UUID

Obtain the previous cluster UUID by running the following command:  esxcli vsan debug object list --all|grep -A1 "(Missing)"

Example output:
Path: /vmfs/volumes/vsan:5282dcbe3a1a60ea-6d92############/00a96f5f-8803-f776-5985-############/Windows_VM.vmdk (Missing)
Path: /vmfs/volumes/vsan:5282dcbe3a1a60ea-6d92############/15e93d63-3244-de33-f502-############/Linux_VM.vmdk (Missing)
Path: /vmfs/volumes/vsan:5282dcbe3a1a60ea-6d92############/9d2d1b63-4021-a5d0-3d35-############/VM129.vmdk (Missing)

The vSAN datastore UUID is just the vSAN Cluster UUID just segmented differently. Below are the steps to segment the vSAN datastore UUID into the vSAN Cluster UUID

• Using the 'UUID' for the vSAN datastore (in the path, example: /vmfs/volumes/vsan:5282dcbe3a1a60ea-6d92############]
  • Copy the vSAN datastore UUID (bold part). To use for the cluster UUID (after changing format). 
• Reformat that ID like this (but with your UUID). [I will use my example]
  • From this: 5282dcbe3a1a60ea-6d92############ to this: 5282dcbe-3a1a-60ea-6d92-############

Step 5 - Rebuild the cluster with the old cluster UUID

Remove the hosts from the vSAN cluster by running command:  esxcli vsan cluster leave
Create the vSAN cluster again using the command from step 3 and the UUID obtained from step 4:  esxcli vsan cluster join -u <UUID>  

Example:
esxcli vsan cluster join -u 5282dcbe-3a1a-60ea-6d92-############

NOTE:  For vSAN ESA, you need to add the flag '-x' to the commands otherwise it will fail with the below error:
vSAN can not be enabled when storagepool disks exist - eui.d77ca3707e2a688a################ ...

Example:
esxcli vsan cluster join -u 5282dcbe-3a1a-60ea-6d92-############ -x

Step 6 - Find your vCenter VM and power it on

Log into the host UI of one of the hosts, register the vCenter VM to the host and power it on. The vCenter Web Client will come online about 10 - 15mins after powering on the vCenter VM.

Sync vCenter and ESXi - if vSAN was using a vDS migrate vSAN traffic back to the vDS

Step 7 - Bring Production back online

Now that vCenter is back online start brig production back online.

If you have any questions or issues re-enabling vSAN please open a case with vSAN Support for further assistance.

Additional Information

You may need to rebuild the unicast table to make the hosts participate in the cluster if they don't automatically join the cluster via Step 3 above.

If the ESXi hosts were added to a new vCenter before the vSAN rebuild, you may see the error unable to enumerate all disks, in that case, follow VMs on vSAN fail to power on with Unable to enumerate all disks

If you accidently clicked on "Turn off vSAN" while attempting to shutdown the cluster and vSAN is now disabled on the cluster see KB Re-enabling vSAN on a cluster when "Turn off vSAN" was used to shutdown the cluster where vCenter resides on the vSAN Datastore