Moving a vSAN Cluster from one vCenter Server to another (= Migrating )

Products

• VMware vSAN 7.x
• VMware vSAN 8.x
• VMware vSAN 6.x

Issue/Introduction

The Steps outlined in this Article can also be performed when the original vCenter is offline permanently or is unable to be restored from Backup.

Impact/Risks:

If performed within the Parameters listed below, the Migration will be seamless.

However, if performed incorrectly, data availability may be temporarily affected until missing or misconfigured Parameters are corrected.

 

Remarks:

• Moving a vSAN File Service enabled cluster from one vCenter to another is not supported, since the data associated with vSAN File Service cannot be migrated
• HCI Mesh Remote Datastores need to be unmounted before the move, and HCI mesh needs to be configured from start in the new vCenter

Environment

vSAN 6.x
vSAN 7.x
vSAN 8.x

Resolution

Please review before proceeding:

• Always maintain current and tested VM backups of critical Virtual Machine workloads
• Review this document in its entirety before proceeding with the migration
• Stretched Cluster: Before migrating a stretch cluster you need to disable the stretch cluster first. To disable Stretched Cluster from Fault Domains (Cluster > Configure > vSAN > Fault Domains > Stretched Cluster > Disable). Once the cluster has been fully migrated then enable Stretched Cluster again
• Stretched Cluster: If it was missed to disable Stretched Cluster in Fault Domains configuration or vCenter failed and had to be redeployed, see KB 326587

 

 

Steps:

1.) Build a new vCenter Server with a Build Version equal or greater than the Version of the ESXi Hosts being migrated
Reason: Version differences can affect the vSAN Skyline Health reporting. 

2.) Prior to migrating the Cluster, please ensure to test the vSAN VMkernel, Host Management, vMotion, and any additional dependent Network routing schemes

 

 

3.) Verify that Networking Configuration Requirements are met

- When using vSphere Standard Virtual Switch no additional steps are required other than ensuring the physical networking routes are working as designed

- When using NSX-T NVDS with Standalone Host configuration, no additional steps are required other than ensuring the physical networking routes are working as designed

Note: If using Transport Node Profile for NSX-T NVDS, then you need to migrate to Standalone Host configuration before proceeding.

- When using vSphere Distributed Switch, see the following documentation to prepare the vDS for migration

Note: While importing vDS settings, do not select "Preserve original distributed switch port group identifiers" as this could negatively impact the migration ( = Reference: Step 4 - Select the Preserve original distributed switch and port group identifiers option )

Moving an ESX/ESXi host with vDS from one vCenter Server to another

Exporting/importing/restoring Distributed Switch configs using vSphere Web Client

Back Up and Restore a vSphere Distributed Switch Configuration

 

4.) Determine if vSAN Data-at-Rest-Encryption is enabled and make certain that Keys and the KMS configurations are well known ( see also: Using Encryption in Your vSphere Environment )

Note: If using a Native Key Provider (NKP) make sure to backup the NKP config.

5.) In the vCenter Web Client: If needed: Create a new Data-Center Inventory Object

6.) In the vCenter Web Client: Create a new Cluster Object with DRS, and vSphere HA, disabled (only enable after vSAN Cluster Hosts are running without issues under the new vCenter ), However EVC should be enabled before bring the hosts into the new cluster, especially if the hosts have mixed CPUs otherwise you will get an error that the host can't join the cluster with powered on VMs without EVC.

 

 

7.) In the Web Client of the new vCenter: Enable vSAN along with the required Services matching the original Cluster e.g. Encryption, Deduplication and/or Compression etc.

If the original vCenter is not available to compare Configurations, the following commands can be used to determine.

 

Deduplication/Compression/Encryption:

localcli vsan storage list |grep -i 'Device\|Is ssd\|Is Capacity Tier\|DEDUPLICATION\|COMPRESSION\|In CMMDS\|ENCRYPTION:' |sed 'N;N;N;N;N;N;s/\n//g' | sort -k9;

Example:

Device: mpx.vmhba0:C0:T1:L0   Is SSD: true   In CMMDS: true   Deduplication: false   Compression: false   Is Capacity Tier: false  Encryption: true

Note: For ESA clusters run localcli vsan storagepool list |grep -i 'Device\|In CMMDS\|Is Encrypted' |sed 'N;N;s/\n//g' |  sort -k9;

 

Encryption enabled (= true)

When using KMS Servers make certain to add the KMS Servers to the newly built vCenter server with the same KMS-Cluster names

and complete all required steps to add the KMS servers into the new vCenter server ( See here for more information: Setting up ).

Note: What was previously called a "Key Management Server cluster" in vSphere 6.5 and 6.7 is now called a "Key Provider".

If an NKP was used then restore the NKP config into the new vCenter.

If assistance is required, please open a Ticket with VMware by Broadcom Support.

Besides the option to view the KMS in the vCenter Web Client, KMS details can also be obtained by logging into one of the Hosts from the vSAN Encryption enabled Cluster via SSH/Putty:

esxcli vsan encryption kms list

esxcli vsan encryption info get

esxcli vsan encryption hostkey get ( Should not be needed for completing the steps in this KB )

esxcli vsan encryption cert get  ( Should not be needed for completing the steps in this KB )

 

Examples:

esxcli vsan encryption kms list

Cluster               Name   KMS Address        Proxy Address  Proxy Port  Username

--------------------  -----  -----------------  -------------  ----------  --------

####-KMS-Cluster      KMS-1  ##.###.##.##:####  N/A            N/A         N/A

 

esxcli vsan encryption info get

Attribute            Value

-------------------  -----

kekId                ########-####-####-####-############

dekGenerationId      1

enabled              True

hostKeyId            ########-####-####-####-############

eraseDisksBeforeUse  False

changing             False

To enable vSAN on an existing cluster:

1. Open the vSphere Web Client.
2. Click the Hosts and Clusters tab.
3. Select the cluster name intended to have vSAN enabled.
4. Validate vSphere HA is disabled on the cluster via Configure > Services > vSphere Availability, if not then temporarily disable it (and re-enable it after enabling vSAN) via Configure > Services > vSphere Availability > Edit > slider to off > OK
5. Click the Configure tab.
6. Under vSAN, select Services, click Turn on vSAN, and then follow the wizard.
7. Click OK.

8.) In the new Web Client of the new vCenter: Create the Virtual Machine vSAN Storage policies that match the vSAN policies from the old Cluster

If the VMs are migrated without matching policies, vSAN may need to perform a resync to ensure the VMs and objects are compliant.

 

If Policy details are unknown due to old vCenter unavailability, the details can also be obtained by logging into one of the Hosts from the vSAN Encryption enabled Cluster via SSH/Putty:

esxcli vsan debug object list | grep -E 'spbmProfileName|spbmProfileId' |sed 'N;s/\n//g'| sort | uniq
      spbmProfileId: 451bcbc3-c25b-4c73-a52a-############      spbmProfileName: FSVM_Profile_DO_NOT_MODIFY
      spbmProfileId: 4b97756b-3c50-481a-a105-############      spbmProfileName: vSAN ESA Default Policy - RAID5
      spbmProfileId: aa6d5a82-1c88-45da-85d3-############      spbmProfileName: vSAN Default Storage Policy

 

 

To obtain more details on the Objects using a specific Policy ( spbmProfileName ), run esxcli vsan debug object list | less

 

Example:

Object UUID: ########-####-####-####-############

Version: 10

Health: healthy

Owner: server.com

Size: 15.00 GB

Used: 0.89 GB

Policy: ** CONTENT BELOW INDICATES POLICY ATTRIBUTES**

cacheReservation: 0

forceProvisioning: 0

spbmProfileId: ########-####-####-####-############

proportionalCapacity: 0

spbmProfileGenerationNumber: 0

stripeWidth: 1

spbmProfileName: vSAN Default Storage Policy

CSN: 16

hostFailuresToTolerate: 1

 

Configuration:

RAID_1

Component: ########-####-####-####-############

Component State: ACTIVE,  Address Space(B): 16106127360 (15.00GB),  Disk UUID: ########-####-####-####-############,  Disk Name: naa.###############:2

Votes: 1,  Capacity Used(B): 486539264 (0.45GB),  Physical Capacity Used(B): 478150656 (0.45GB),  Host Name:server.com

Component: ########-####-####-####-############

Component State: ACTIVE,  Address Space(B): 16106127360 (15.00GB),  Disk UUID: ########-####-####-####-############,  Disk Name: naa.###############:2

Votes: 1,  Capacity Used(B): 486539264 (0.45GB),  Physical Capacity Used(B): 478150656 (0.45GB),  Host Name:server.com

Witness: ########-####-####-####-############

Component State: ACTIVE,  Address Space(B): 0 (0.00GB),  Disk UUID: ########-####-####-####-############,  Disk Name: naa.###############:2

Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name:server.com

Type: vdisk

Path: /vmfs/volumes/vsan:###############-########-##############/########-####-####-####-############/VCSA####_6.vmdk (Exists)

Group UUID: ########-####-####-####-############

Directory Name: N/A

 

9.) Unicast Communication
In vSAN 6.6 and above, it is critical to maintain vSAN Virtual Networking Unicast communication between vSAN Hosts on the vSAN Network ( If needed see KB 326427 for more details ) .

On each vSAN Host to be migrated, log into via SSH/Putty and run the following command:

esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListUpdates

 

This will ensure that the Hosts when disconnected from the old vCenter will not lose their Unicast configuration and therefore will maintain their vSAN connectivity.

If this step is skipped, the Unicast address table will be cleared and Data Availability will be affected until the Table is repopulated by the new vCenter. 

 

 

10.) Number of Hosts in the Cluster

Log into one of the vSAN Hosts via SSH/Putty and run: esxcli vsan cluster get

Make a note of the "Sub-Cluster: Member Count" to ensure that the number of Hosts belonging to this Cluster will be the same after the Migration to the new vCenter has been completed.

 

 

11.) Skip if old vCenter is unavailable: In the Web Client: Disconnect one vSAN Host by right-clicking the Host and select "Disconnect"

12.) Skip if old vCenter is unavailable: In the Web Client: Right-Click the vSAN Host from 11.) and select "Remove from Inventory" (Before doing so ensure you have reviewed KB 337625 )

 

 

13.) New vCenter: Web Client: Add the vSAN Host which was just removed by placing it outside of the vSAN Cluster 

Note: If the Host(s) is being placed directly into the vSAN Cluster, it will be forced into Maintenance Mode which can negatively impact running VMs

14.) Once the Task from step 13.) completed, move the vSAN Host into the vSAN Cluster (wait for Task to complete)

15.) Check on the Host which was just moved into the Cluster

Log into via SSH/Putty and run: esxcli vsan cluster get

Make a note of the "Sub-Cluster: Member Count".

It should reflect the number of Hosts showing in the Web Client under the vSAN Cluster.

If there is a mismatch, then the newly added Hosts cannot communicate to the other Hosts which have not been added to the new vCenter yet.

Proceed with double-checking Steps 3, 9, 13, 14 (including that any related tasks are showing as "completed" ).

Ensure that vSAN Hosts can communicate via the vSAN VMKernel Port with each other ( vSAN Network Troubleshooting )

If the cause of the mismatch cannot be found, do not proceed with the next steps, instead proceed with opening a Ticket with VMware by Broadcom Support.

 

16.) Proceed only if no mismatch was found and/or the mismatch got resolved (= Step 15): Repeat Steps 13 - 15 with the next vSAN Host

17.) Once all Hosts have been successfully added to the new Cluster via Steps 13-15:

On each vSAN Host, log into via SSH/Putty and run the following command:

esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates

 

 

18.) Verify Host and VM Connectivity by pinging the new vCenter from the Management Interface on each Host (= usually vmk0)

19.) In the Web Client of the new vCenter: Apply the vSAN Storage Policies to the VMs which were created via Step 8.) 

Note:

Assuming the new Policy is consistent with the old Policy and all the Hosts have remained in a healthy state, a Resync shouldn't be triggered.

It is recommended to assign the new Policy not to all VMs at the same time. Work in batches and wait for the Tasks to complete.

This is considered as Best Practice to control the I/O Load on the Cluster caused by any triggered Resync activity.

If there is any Resync, wait for it to complete before proceeding with assigning the Policy to remaining VMs.

 

 

 

 

20.) In the Web Client of the new vCenter: Menu --> Policies & Profiles --> VM Storage Policies
Check VM Compliance as shown below. All VMs should be shown.
( Check Compliance for a VM Storage Policy )