How to create a Stretched cluster on a NSX-T Workload Domain in VMware Cloud Foundation 3.8
search cancel

How to create a Stretched cluster on a NSX-T Workload Domain in VMware Cloud Foundation 3.8

book

Article ID: 336762

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

Deploying a VSAN stretched Cluster on a NSX-T Workload Domain is not a functionality currently available in VMware Cloud Foundation. 
Below are the prerequisites required to successfully complete this workflow.

  1. Enough VSAN and ESXi licenses : VSAN and ESXi licenses ( CPUs) must be adequate
  2. Naming conventions:
    • Host group : {Cluster-Name}_primary-az-hostgroup , {Cluster-Name}_secondary-az-hostgroup
    • VM group: {Cluster-Name}_primary-az-vmgroup
    • Preferred site: {Cluster-Name}_primary-az-faultdomain
    • Secondary Site: {Cluster-Name}_secondary-az-faultdomain
  3. Number of ESXi hosts on preferred site/primary-az-hostgroup should be equal to Number of hosts on secondary site/secondary-az-hostgroup.


Symptoms:

The article below describes how to deploy NSX-T based Workload domain with stretch cluster where the management, vMotion, vSAN, and vTEP networks are Layer 2 stretched between both availability zones. While this deployment model is supported it is not the recommended and preferred deployment model.

The workflow should only use this Layer 2 stretch deployment model after extensively investigating and ruling out the standard deployment model where the management, vMotion, VSAN, and vTEP networks separate Layer 2 networks per site


Environment

VMware Cloud Foundation 3.8.x

Resolution

Deploy and configure Witness host:

Once bringup is completed, access the vCenter Server associated to Management Domain and create following artifacts
  1. Create a new cluster.
    • Create a DVS switch with a portgroup "MGMT-PG" and tag the portgroup with the Default Management VLAN.
    • Create a portgroup "VSAN-PG" on the VDS and tag it with the Witness VLAN. This portgroup will be used for VSAN traffic for the witness host.
    • Add an ESXi Host, which is not part of any cluster to the new cluster.
    • Associate host to to the DVS
  2. Deploy the VSAN WITNESS OVA on the new cluster. When in the deployment wizard on the "Select Networks" section, associate "VSAN-PG" to Witness Network and "MGMT-PG" for Management Network and provide appropriate root passwords.
  3. Once the Witness VM is deployed, access web console of the Witness VM and configure Management network and DNS appropriately and enable SSH.
  4.  Add the Witness VM as a host to the "Workload vCenter" as a standard host at the Datacenter level, it should not be part of any cluster.
  5. Once Witness is added as host, access the Host in the vCenter Web Client and navigate to Configure ->Networking -> VMKernel Adapter. Configure the IP address details for vmk1 associated to Witness Switch. 
  6. Provide IP routing rules for the VSAN networks for the hosts in AZ1 and AZ2.

Create NSX-T enabled domain/cluster

Using VCF SDDC Manager UI, create a NSX-T Workload Domain or create a new cluster in an existing NSX-T Workload Domain.

Note: Ensure that the cluster has sufficient ESXi hosts such that VSAN Stretch cluster operation can be performed.
 

Expand NSX-T enabled cluster

NSX-T enabled cluster is expanded using the "Add Host" functionality in SDDC Manager. Use the steps below to perform expansion of NSX-T cluster with AZ2 ESXi hosts.

  1. Commission the AZ2 ESXi hosts using the same network pool.
  2. In the SDDC Manager UI, navigate to the NSX-T Workload Domain and select the cluster to expand. Click Actions and select Add Host and choose AZ2 hosts and complete finish cluster expansion.

Create fault domains and other stretch cluster related configurations

Complete the below steps to enable a stretched VSAN Cluster.

  1. Access vCenter Server Web Client associated with the desired cluster.
    • Navigate to the vSAN cluster.
    • Click the Configure tab.
    • Under vSAN, click Fault Domains.
    • Click the Stretched Cluster Configure button to open the stretched cluster configuration wizard.
    • Select the hosts or the fault domain that you want to assign to the secondary domain and click >>.The hosts that are listed under the Preferred fault domain are in the preferred site.
    • Click Next.
    • Select a witness host that is not a member of the vSAN stretched cluster and click Next.
    • Claim the storage devices on the witness host and click Next.
    • Select one flash device for the cache tier, and one or more devices for the capacity tier.
    • On the Ready to complete page, review the configuration and click Finish.
  2. Navigate to Configure -> Services and complete the below configurations for the cluster.
    • In vSphere Availability, Set Admission Control, the Host failures cluster tolerates  needs to be set to 50% of the cluster. If the stretched cluster is a total of  8 hosts this value will be 4.
    • In vSphere Availability -> Advanced Options, set VSAN gateway as isolated address as following:  das.isolationaddress1=<VSAN-GATEWAY-IP-for-preferred-site-hosts>

    • Navigate to VM/Host Groups and click Add to create two host groups, one for the preferred site and one for the secondary site.

    • In VM/Host Groups create two VM groups, one to hold the VMs on the preferred site and one to hold the VMs on the secondary site.
    • In VM/Host Rules the HA settings should respect VM-Host affinity rules during failover.
    • Navigate to Policies and Profiles and edit the vSAN profile associated with the cluster. In the vSAN panel, under the Availability tab, set the following:
      •  Site disaster tolerance to Dual Site Mirroring (stretched cluster)
      • Failure to tolerate is set to 1 failure - Raid-1  (Mirroring).
  3. Download setup patch vcf-stretch-cluster-patch.zip attached to this article.
  4. Using a file transfer utility, copy the file to the /tmp/ directory on the SDDC Manager VM.
  5. SSH to the SDDC Manager using the vcf user and change to root:
su -
  1. Unzip the file:
unzip /tmp/vcf-stretch-cluster-patch 
  1. Change directory to the extracted folder:
cd /tmp/vcf-stretch-cluster-patch 
  1. Execute the setup.sh script to update the SOS package.
  2. After completing the stretch cluster operation, execute the below command to enable the stretch cluster flag for in the SDDC Manager inventory.The below command will enable the stretch flag and will block the Add Host functionality.
/opt/vmware/sddc-support/sos ---enable-stretch-cluster-flag  {cluster-name} --domain-name {Workload-Domain-name}
 

Day2 operations on a Stretch Cluster

  1. Adding host to the stretched cluster.
  2. Removal of an ESXi host.
  3. Upgrading the stretched cluster through LCM.

ESXi Hosts can be added through the Add Host workflow in the SDDC Manager UI.

  1. The stretched cluster flag, firstly, needs to be disabled to allow the Add Host functionality to be enabled in the SDDC Manager UI. Please run the below command on the SDDC Manager using root:

/opt/vmware/sddc-support/sos --disable-stretch-cluster-flag {cluster-name} --domain-name {Workload-Domain-name}

  Once a host is added to a stretched cluster, follow the below instructions to make the host part of the fault domains.

  1. Once host is added to the stretched cluster, in the vCenter Web Client the cluster will have a critical alarm for  "VSAN Health Alarm "Unexpected number of fault domains". This is generated as the ESXi host that was added is not part of any of the fault domains. 
  2. Select Cluster -> Configuration -> Fault Domains.
  3. Select the ESXi host, which is not part of any of the fault domain, click on the "Move to fault domain" icon and select the fault domain the ESXi host should be part of.
  4. Re-enabled the stretched cluster flag:
/opt/vmware/sddc-support/sos ---enable-stretch-cluster-flag  {cluster-name} --domain-name {Workload-Domain-name}

Upgrading Stretch Cluster

ESXi hosts that are part of the SDDC Manager inventory, except VSAN Witness Host, will be upgraded through SDDC Manager's LCM functionality. 

NSX-T upgrade will also be handled by through SDDC Manager's LCM functionality. 

Upgrade the vSAN Witness Host

The Witness host upgrade is performed outside of the SDDC Manager as the witness host is not part of the inventory. The upgrade of witness host need to be performed using the vCenter Update Manager.

Below are the steps to be performed for upgrade of the witness. 

  1. Login to vCenter Web Client, where witness host is added.
  2. Navigate to Home -> Update Manager
  3. Select the ESXi Images tab and click IMPORT to upload the ESXi image.
  4. Select the Baselines tab and click NEW -> Baseline. Provide a name for the baseline and ensure Content is Upgrade. In Select Image, use the uploaded ESXi Image from Step 3. Click OK.
  5. Navigate to Hosts and Clusters, select the Witness host and select the Updates tab.
  6. Attach the baseline to the host and click on Remediate. After this task is successful, the host will have been upgraded to the desired version.
 

Remove of an ESXi host from the Stretched Cluster.

Should a host be required to be removed from the cluster due to any reason, the Remove Host functionality in the SDDC Manager UI should be used to properly hand its removal from vCenter and the cluster.


Additional Information

Impact/Risks:
Note: This workflow is not intended for VMware Cloud Foundation on VxRail deployments. To stretch VCF on VxRail deployments please refer to the KB How to create a Stretched cluster on a NSX-T Workload Domain on VMware Cloud Foundation on VxRail 3.8 environment

Attachments

vcf-stretch-cluster-patch get_app
Powered by Wolken Software