How to create a Stretched cluster on a NSX-T Workload Domain on VMware Cloud Foundation on VxRail 3.8 environment

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:
Theis article describes how to deploy a NSX-T based Workload domain on a VMware Cloud Foundation on VxRail 3.8 environment with a stretch cluster where the management, vMotion, vSAN, and vTEP networks are Layer 2 stretched between both availability zones. While this deployment model is supported it is not the recommended and preferred deployment model.

Prerequisite
Deploying a VSAN stretched Cluster on a NSX-T Workload Domain is not a functionality currently available in VMware Cloud Foundation on VxRail. Below are the prerequisites required to successfully complete this workflow.

1. Enough VSAN and ESXi licenses : VSAN and ESXi licenses ( CPUs) must be adequate
2. Naming conventions:
   • Host group : {Cluster-Name}_primary-az-hostgroup , {Cluster-Name}_secondary-az-hostgroup
   • VM group: {Cluster-Name}_primary-az-vmgroup
   • Preferred site: {Cluster-Name}_primary-az-faultdomain
   • Secondary Site: {Cluster-Name}_secondary-az-faultdomain
3. Number of ESXi hosts on preferred site/primary-az-hostgroup should be equal to number of hosts on secondary site/secondary-az-hostgroup.
4. Complete bringup of the environment following the Administration Guide.

Environment

VMware Cloud Foundation on VxRail 3.8.x

Resolution

Deploy a NSX-T workload domain 
To deploy a NSX-T workload domain and add required clusters follow these steps.

Deploy and configure Witness host

1. Deploy the Witness appliance following the VSAN documentation on an environment outside the VMware Cloud Foundation on VxRail rack but with access to the same management and vsan network. 
2. Once the Witness appliance is deployed, access web console of the Witness Appliance and configure Management network and DNS appropriately and enable SSH.
3. Add the Witness Appliance as a host to the "Workload vCenter" of the VMware Cloud Foundation on VxRail environment as a standard host at the Datacenter level, it should not be part of any cluster.
4. Once Witness is added as host, access the Host in the vCenter Web Client and navigate to Configure ->Networking -> VMKernel Adapter. Configure the IP address details for vmk1 associated to the proper DVS. 
5. Provide IP routing rules for the VSAN networks for the hosts in AZ1 and AZ2.

Expand a NSX-T enabled cluster
NSX-T enabled cluster is expanded using the expansion procedure from the SDDC Manager. Ensure there are even number of hosts (minimum 6 ) in the cluster so that they can be divided up into AZ1 and AZ2.

Create fault domains and other stretch cluster related configurations
Complete the below steps to enable a stretched VSAN Cluster.

1. Access vCenter Server Web Client associated with the desired cluster. 
   • Navigate to the vSAN cluster.
   • Click the Configure tab.
   • Under vSAN, click Fault Domains.
   • Click the Stretched Cluster Configure button to open the stretched cluster configuration wizard.
   • Select the hosts or the fault domain that you want to assign to the secondary domain and click >>.The hosts that are listed under the Preferred fault domain are in the preferred site.
   • Click Next.
   • Select a witness host that is not a member of the vSAN stretched cluster and click Next.
   • Claim the storage devices on the witness host and click Next.
   • Select one flash device for the cache tier, and one or more devices for the capacity tier.
   • On the Ready to complete page, review the configuration and click Finish.
2. Navigate to Configure -> Services and complete the below configurations for the cluster. 
   • In vSphere Availability, Set Admission Control, the Host failures cluster tolerates needs to be set to 50% of the cluster. If the stretched cluster is a total of  8 hosts this value will be 4.
   • In vSphere Availability -> Advanced Options, set VSAN gateway as isolated address as following:  das.isolationaddress1=<VSAN-GATEWAY-IP-for-preferred-site-hosts>
   • Navigate to VM/Host Groups and click Add to create two host groups, one for the preferred site and one for the secondary site.
   • In VM/Host Groups create two VM groups, one to hold the VMs on the preferred site and one to hold the VMs on the secondary site.
   • In VM/Host Rules the HA settings should respect VM-Host affinity rules during failover.
   • Navigate to Policies and Profiles and edit the vSAN profile associated with the cluster. In the vSAN panel, under the Availability tab, set the following:
     • Site disaster tolerance to Dual Site Mirroring (stretched cluster)
     • Failure to tolerate is set to 1 failure - Raid-1  (Mirroring).

Day-N operations on a Stretched Cluster

1. Adding host to the stretched cluster.
2. Removal of a host from the stretched cluster.
3. Upgrading the stretched cluster through LCM.

Additional ESXi Hosts can be added through the Add Host workflow in the SDDC Manager by following these steps.

Once a host is added to a stretched cluster, follow the below instructions to make the host part of the fault domains.

• Once host is added to the stretched cluster, in the vCenter Web Client the cluster will have a critical alarm for  "VSAN Health Alarm "Unexpected number of fault domains". This is generated as the ESXi host that was added is not part of any of the fault domains. 
• Select Cluster -> Configuration -> Fault Domains.
• Select the ESXi host, which is not part of any of the fault domain, click on the "Move to fault domain" icon and select the fault domain the ESXi host should be part of.

Upgrading Stretch Cluster

ESXi hosts that are part of the SDDC Manager inventory, except VSAN Witness Host, will be upgraded through SDDC Manager's LCM functionality. 

NSX-T upgrade will also be handled by through SDDC Manager's LCM functionality. 

Upgrade the vSAN Witness Host

The Witness appliance upgrade is performed manually outside of the SDDC Manager.