Moving a vSAN cluster from one vCenter Server to another
Article ID: 326849
Updated On:
Products
Issue/Introduction
This article explains how to move a vSAN cluster from one vCenter Server to another vCenter Server.
Impact/Risks:
Notes:
- Moving a vSAN File Service enabled cluster from one vCenter to another is not supported, since the data associated with vSAN File Service cannot be migrated.
- HCI mesh remote datastores need to be unmounted before the move, and need HCI mesh needs to be configured from start in the new vCenter.
Environment
VMware vSAN 6.x
VMware vSAN 7.0.x
VMware vSAN 8.0.x
Resolution
The following steps can be followed when migrating vSAN clusters from one vCenter to another vCenter and when the original vCenter is offline permanently or unable to be restored from backup.
Notes:
- Always maintain current and tested VM backups of critical Virtual Machine workloads.
- Review this document in its entirety before proceeding with the migration.
- For stretched clusters, to disable stretched cluster from Fault Domains (Cluster > Configure > vSAN > Fault Domains > Stretched Cluster > Disable) prior to migrating the cluster to the new vCenter. Once the cluster has been fully migrated then enable stretched cluster again.
- If forgot to disable stretched cluster in Fault Domains configuration or vCenter failed and had to be redeployed, see KB 77163
Base Steps:
1. Build a new vCenter server with a build version equal or greater than the version of the ESXi Hosts being migrated.
Version differences may affect the Skyline vSAN Health services in rare cases.
2. Prior to migrating the cluster, please be sure to test the vSAN VMkernel, Host Management, vMotion, and any additional dependent network routing schemes.
3. Determine if the host networking is configured with vDS, vSS, or NSX-T NVDS virtual switches.
- When using vSphere Standard Virtual Switch no additional steps are required other than ensuring the physical networking routes are working as designed.
- When using vSphere Distributed Switch, please reference further material/instruction KB links below to prepare the vDS for migration.
Note: While importing vDS settings, do not select preserve original distributed switch port group identifiers as this could negatively impact the migration.
- When using NSX-T NVDS with standalone host configuration, no additional steps are required other than ensuring the physical networking routes are working as designed.
Note: If using Transport Node Profile for NSX-T NVDS, then need to migrate to standalone host configuration.
For more information on the migration of Hosts, see Moving an ESX/ESXi host with vDS from one vCenter Server to another
For more information on migration of vDS see Exporting/importing/restoring Distributed Switch configs using vSphere Web Client
Backing Up and Restoring a vSphere Distributed Switch Configuration
For more information on vSphere see https://docs.vmware.com/
- Once the cluster networking configuration requirements are satisfied, then may proceed with the next steps of the migration.
4. Determine if vSAN Data at Rest Encryption is in use and make certain that keys and the KMS configurations are well known. See the guide for further guidance in preparing the cluster for migration.
Also, See further details in this instruction on step 7a.
Document: Using Encryption on a vSAN Cluster
5. Create a new cluster with DRS, vSphere HA, and EVC Disabled. These options should only be enabled after the cluster is moved and stable on the new vCenter.
6. Create a new cluster Inventory object.
7. a. Enable vSAN along with required services matching the original cluster parameters such as Encryption and or Deduplication and Compression.
- If the original vCenter is not available to compare configurations, the following command can be used to list the vSAN services in use.
localcli vsan storage list |grep -i 'Device\|Is ssd\|Is Capacity Tier\|DEDUPLICATION\|COMPRESSION\|In CMMDS\|ENCRYPTION:' |sed 'N;N;N;N;N;N;s/\n//g' | sort -k9;
localcli vsan storage list |grep -i 'Device\|Is ssd\|Is Capacity Tier\|DEDUPLICATION\|COMPRESSION\|In CMMDS\|ENCRYPTION:' |sed 'N;N;N;N;N;N;s/\n//g' | sort -k9; Device: mpx.vmhba0:C0:T1:L0 Is SSD: true In CMMDS: true Deduplication: false Compression: false Is Capacity Tier: false Encryption: true Device: mpx.vmhba0:C0:T2:L0 Is SSD: true In CMMDS: true Deduplication: false Compression: false Is Capacity Tier: true Encryption: true b. When KMS Encryption service is in use -
Make certain to add the KMS servers to the newly built vCenter server with the same KMS-Cluster names and complete all required steps to add the KMS servers into the new vCenter server.
-
- If it is not sure about the KMS cluster procedures, please reference KMS documentation and/or contact VMware Global Support to consult.
- If it is unsure about the KMS cluster name and the original cannot be recovered, to obtain the KMS details about KMS-cluster info by using "esxcli vsan encryption kms list" and "esxcli vsan encryption info get" on any of the hosts in the cluster, where vSAN encryption was enabled. Other additional information such as host-key and certs can be fetched using "esxcli vsan encryption hostkey get" and "esxcli vsan encryption cert get".
Example:
# esxcli vsan encryption kms list Cluster Name KMS Address Proxy Address Proxy Port Username -------------------- ----- ----------------- ------------- ---------- -------- ####-KMS-Cluster KMS-1 ##.###.##.##:#### N/A N/A N/A # esxcli vsan encryption info get Attribute Value ------------------- ----- kekId ########-####-####-####-############ dekGenerationId 1 enabled True hostKeyId ########-####-####-####-############ eraseDisksBeforeUse False changing False
8. Create the Virtual Machine vSAN storage policies that match the vSAN policies from the old cluster. Ensure that rulesets have matching policies. If the VMs are migrated without matching policies, vSAN may need to perform a resync to ensure the VMs and objects are compliant.
If the policy names and policy attributes were unknown due to old vCenter unavailability, the following commands can be used to obtain the policy settings:"esxcli vsan debug object list | grep spbmProfileId | sort | uniq" or "esxcli vsan debug object list | grep spbmProfileName | sort | uniq"
Review the attributes for each object using the policies by running "esxcli vsan debug object list |less" and search by the policy name (found above).
Example :
# esxcli vsan debug object list | grep spbmProfileId | sort | uniq
spbmProfileId: ########-####-####-####-############ spbmProfileId: ########-####-####-####-############
# esxcli vsan debug object list | grep spbmProfileName | sort | uniq
spbmProfileName: VM Storage Policy-DN spbmProfileName: vSAN Default Storage Policy
An example is provided below of the policy attributes for the spbmProfileName: vSAN default storage policy
Run Command: esxcli vsan debug object list | less
Example:
Object UUID: ########-####-####-####-############ Version: 10 Health: healthy Owner: server.com Size: 15.00 GB Used: 0.89 GB Policy: ** CONTENT BELOW INDICATES POLICY ATTRIBUTES** cacheReservation: 0 forceProvisioning: 0 spbmProfileId: ########-####-####-####-############ proportionalCapacity: 0 spbmProfileGenerationNumber: 0 stripeWidth: 1 spbmProfileName: vSAN Default Storage Policy CSN: 16 hostFailuresToTolerate: 1
Configuration:
RAID_1 Component: ########-####-####-####-############ Component State: ACTIVE, Address Space(B): 16106127360 (15.00GB), Disk UUID: ########-####-####-####-############, Disk Name: naa.###############:2 Votes: 1, Capacity Used(B): 486539264 (0.45GB), Physical Capacity Used(B): 478150656 (0.45GB), Host Name:server.com Component: ########-####-####-####-############ Component State: ACTIVE, Address Space(B): 16106127360 (15.00GB), Disk UUID: ########-####-####-####-############, Disk Name: naa.###############:2 Votes: 1, Capacity Used(B): 486539264 (0.45GB), Physical Capacity Used(B): 478150656 (0.45GB), Host Name:server.com Witness: ########-####-####-####-############ Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: ########-####-####-####-############, Disk Name: naa.###############:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name:server.com
Type: vdisk Path: /vmfs/volumes/vsan:###############-########-##############/########-####-####-####-############/VCSA####_6.vmdk (Exists) Group UUID: ########-####-####-####-############ Directory Name: N/A
9. In vSAN 6.6 and above, it is critical to maintain vSAN virtual networking unicast communication between hosts on the vSAN network. Please see KB for configuration details.
Configuring vSAN Unicast networking from the command line
10. Set the following configuration via CLI on each ESXi vSAN Cluster Member Host to be migrated.
esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListUpdates
This will ensure when the hosts are disconnected from the original vCenter, they will not lose their unicast configurations and therefore their connectivity between hosts. If this step is skipped, the unicast address table will be cleared and data availability will be affected temporarily until the table is manually repopulated.
11. Run the command "esxcli vsan cluster get" from any hosts in the cluster and take note of the host count to ensure the count matches before and after the migration.
12. Disconnect one host from vCenter by right clicking the host and using the disconnect option. (**Not applicable if old vCenter is unavailable)
13. Remove the host from inventory. Make sure to follow KB 337625 if using a vDS (**Not Applicable if old vCenter is unavailable)
14. Add the hosts only to the DataCenter level of the new vCenter inventory. Do NOT add the hosts directly to the new vSAN enabled cluster otherwise, the host will be forced into Maintenance Mode. (Avoiding Maintenance Mode is key for a seamless downtime migration.)
15. Once the host is registered in the new vCenter inventory, under the DataCenter Inventory Object, it can safely be dragged and dropped into the new vSAN enabled cluster from within the vCenter UI.
16. Before moving on to the next host, verify there is host-to-host vSAN communication. Run the command "esxcli vsan cluster get" from ESXi CLI and take note of the host count to ensure the count matches.
It is good to practice this command each time a host is disconnected, moved, and connected. If the host is not communicating, i.e. the host count is less than expected then it is highly likely a critical step has been missed. Please review the steps and ensure each host's unicast tables are complete and host-to-host communication on the vSAN network is fully functional.
To verify unicast agent table, use below command -
esxcli vsan cluster unicastagent list
17. Once the first host is moved and connectivity checks are performed, move forward with the remaining hosts. Please be certain to follow the steps carefully and check the cluster membership along the way to prevent any data availably impacts.
18. Once all hosts have been added to the new cluster and access to the VMs has been verified, revert the hosts' unicast table configuration. This procedure should be completed at the completion of the migration, and before the hosts are rebooted next. Host rebooting is not necessary for the procedure's success.
esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates
19. Again verify host and VM connectivity again by pinging the new vCenter from the management interface on each host.
20. Apply the vSAN storage policy to the VMs.
Click OK. Assuming the policy is consistent with the old policy and all the hosts have remained in a healthy state, a resync shouldn't be triggered. It is recommended to perform this action, one at a time. The goal is to move gradually to prevent creating a major resync in the cluster.
21. Repeat this action for the remaining VMs. When performed correctly, there will be no reconfigurations/resyncs. If the VMs start resyncing then there is an inconsistency in the ruleset from the original to the new storage policies. Please allow the resync to complete before proceeding.
22. Verify by browsing to Menu, Policies and Profiles, VM Storage Policies, desired storage policy, and VM Compliance. All the VMs should be shown.
Additional Information
Feedback
