How to enable EVC in vCenter Server 6.5/6.7/7.0/8.0 if VC VM is part of the same vSAN cluster

Products

• VMware vSAN

Issue/Introduction

This article provides steps to enable Enhanced vMotion Capability (EVC) on a vSAN cluster when vCenter Server is installed on a virtual machine running in the cluster.

Symptoms:

• While enabling EVC mode for a cluster, hosts are reported with below compatibility status:

The host cannot be admitted to the cluster's current Enhanced vMotion Compatibility mode. Powered-on or suspended virtual machines on the host may be using CPU features hidden by that mode.
 

• vCenter Server Virtual Machine managing the cluster is part of the same cluster

Environment

VMware vSAN 7.0.x
VMware vSAN 6.7.x
VMware vSAN 6.6.x
VMware vSAN 8.0.x

Resolution

Precheck :

1. It is highly recommended to have a full backup of the vCenter Server and there should not be any snapshots prior to this activity
2. Check vSAN Health ensuring all health checks are reporting Green/Healthy
   • vCenter Web Client > Host and Cluster View > vSAN Cluster  > Skyline Health > ( Run test and wait for 5 minutes ) 
3. All the VMs in the vSAN cluster must be set with a policy with protection FTT=1 ( ex. vSAN Default Storage Policy) or higher. All the VMs must be storage policy compliant 
   • vCenter Web Client > Host and Cluster View > VM > Config > Storage Policy 
4. The vCenter Server must be running through Standard Switch to communicate with the external world and at least two ESXi hosts in the vSAN cluster should be configured with Standard Switch where the customer can migrate the vCenter Server on each of them and validate if the Web-Client is working in the normal manner.
5. If the ESXI host is not rebooted for a very long time ( more than 180 days ), it is advisable to reboot one host at a time in a rolling reboot manner to ensure all the state vmkerenel/hostd processes are cleaned and no new unforeseen hardware - software errors are detected during the activity. The host must be put in Maintenamce mode of Ensure Accessibility ( default Maintenamce Mode )  before reboot.

To enable EVC on a vSAN cluster when vCenter Server is installed on a virtual machine running in the cluster:

1. Connect to vCenter Server using the vSphere Web Client.
2. Right-click a data center in the inventory and select New Cluster.
3. Type a name for the cluster.
4. Expand EVC and select a baseline CPU feature set from the EVC mode drop-down
5. Enable vSAN with all the same feature settings as the existing cluster 
6.  SSH to all member hosts in the old vSAN cluster, invoke the below command to disable vSAN unicast update
   • esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListUpdates
7. Select 1st host in the cluster, migrate all powered-on VMs owned by it to other hosts in the cluster
8. Put the host into Maintenance Mode with No data migration
9. Right-click the ESXi host, select Connection, and then Disconnect.
10. Drag and drop the ESXi host into the new EVC vSAN cluster
11. Right-click on the host, select Connection, and then Connect.
12. Exit from Maintenance Mode
13. Using the VMware Host Client, directly connect to the ESXi host that is hosting the vCenter Server virtual machine.
14. Right-click the vCenter Server virtual machine and click Edit Settings.
15. Click the VM Options tab.
16. Click General Options.
17. Make a note of the host (vCenter virtual machine is running), location, and name of the virtual machine configuration file (.vmx) on the datastore. This information is required in step 18 and later.
18. Power off the vCenter Server virtual machine.
19. Connect to the host noted in step 13 using the Host Client.
20.  Right-click the vCenter Server virtual machine and click Unregister.
21.  Using the Host Client, connect directly to the ESXi host that is in the EVC cluster.
22.  Browse the datastore that contains the virtual machine configuration file for the vCenter Server virtual machine as noted in step 13.
23. Right-click the virtual machine configuration file and click Register VM.
24. Power on the vCenter Server virtual machine.
25. Using the vSphere Web Client, connect to the vCenter Server.
    • You now have vCenter Server running in a virtual machine on an ESXi host that is in an EVC cluster. All other virtual machines are running on ESXi hosts that are outside of the EVC cluster.
26. To add the ESXi hosts that are outside the EVC vSAN cluster into the EVC vSAN cluster, move the virtual machines from the hosts.
    • Note: You can attempt to migrate those virtual machines (while powered on) to an ESXi/ESX host that is already in the EVC vSAN cluster. If this migration fails (for example, due to the EVC baseline configuration), power off the virtual machines and then migrate them to an ESXi host in the EVC vSAN cluster.
27. After all the virtual machines are moved from the ESXi host, right-click the host and click Disconnect to disconnect it from the vCenter Server Inventory.
28. Drag and drop the disconnected host into the EVC vSAN cluster.
29. Right-click the ESXi host and click Connect to connect it to the vCenter Server Inventory.
30. Repeat steps 24-27 for each ESXi host until all hosts are part of the EVC vSAN cluster.
31. SSH to all hosts in the cluster, invoke the below command to re-enable the vSAN unicast update
    • esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates
32. Check vSAN health check on the EVC vSAN cluster, make sure there is no network partition reported; Also check "vCenter state is authoritative" check item, if it reports any error, click the remediation button to force update vSAN unicast generation ID.

For a stretched/2-node stretch cluster, follow the below steps:

1. Follow steps 1-5 above but don't enable Stretched cluster just yet, this will be done at a later step
2. Create an FTT0 Storage Policy (for 2-node stretch clusters only)
3. Set this policy to all VM namespace folders (for 2-node stretch clusters only)
4. Follow steps 6-30 above
5. Manually remove witness uncast entry from all data nodes with the following command: 
   • esxcli vsan cluster unicastagent remove -a <Witness_VSAN_IP>
6. Deploy a new witness appliance
7. Enable Stretched cluster using the newly deployed witness host

Note: Steps 2 & 3 solves a problem where the vswp object cannot be created when you power on the VM on the destination host during cluster migration. You can set the VM namespace folder back to FTT1 once the cluster migration is complete.

Note: In step 5 vSAN data will become non-compliant due to the lack of a witness component.

Additional Information

Impact/Risks:
Caution: VMware does not recommend using these steps for a vCenter Server virtual machine that is running on a Virtual Distributed Switch (VDS). Move the vCenter Server virtual machine to a standard vSwitch or to an ephemeral port group on the distributed switch before completing the steps outlined in this article. More information on ephemeral port groups can be found in KB Static (non-ephemeral) or ephemeral port binding on a vSphere Distributed Switch (1022312)

Notes: 

• The below steps as outlined in the Resolution section is risky and VMware recommends that EVC should be considered and enabled for your vSAN cluster prior to it being put into production. Also, the below steps are for vSAN 6.6 and higher.
• These steps should not be attempted if VxRail or VCF is in use.
• These steps should not be attempted if vSAN File Service is in use.