'vSAN cluster configuration consistency' alert after moving vSAN ESA hosts from one cluster to another (Cluster migration)

Products

• VMware vSAN

Issue/Introduction

This article is about the vSAN Skyline Health alert 'vSAN cluster configuration consistency' which may be triggered after performing a migration activity of vSAN ESA hosts from one cluster to another. (Moving vSAN Hosts/Nodes from one cluster to another within the same vCenter or Moving a vSAN Cluster from one vCenter Server to another (= Migrating ))

Symptoms:

• Post migrating vSAN ESA hosts, which had data-at-rest encryption enabled, to a new cluster the vSAN Skyline Health alert "vSAN Cluster configuration consistency" gets triggered, with the following issues reported:

  • "Data is encrypted with an out of date Data Encryption Key"

  • "Object data encryption key is inconsistent with the cluster or host side encryption configuration"

  • "Data Encryption Key is encrypted with an out of date Key Encryption Key"

  • "Host key for host core dump encryption is inconsistent with cluster configuration"

  • "Data Encryption Key is corrupted"

• There is no issues with connectivity of KMS with vCenter or the ESXi host.

• Even though the target vSAN cluster had data-at-rest encryption enabled before the hosts were moved, this error is observed.

• Clicking on "Remediate" button on the health checks would result in the error "A general system error occurred: unexpected deep rekey" and the alerts remain.

• The vSAN capacity devices in the storagepool remain mounted and encrypted on all hosts.

Environment

VMware vSAN 8.x (ESA)

Cause

• For vSAN ESA, just enabling encryption on the target cluster and moving the hosts to it is not the right way to migrate the hosts.

• Prior to moving the hosts to the target cluster, the encryption configuration from the hosts must be pulled and applied to the target vSAN cluster.

• This cannot be performed via the UI, but rather requires a custom script provided by VMware vSAN Engineering Team to be run to configure the encryption configuration to the target vSAN cluster.

Resolution

.To resolve this issue, the hosts need to be migrated to a new vSAN ESA cluster with the below steps:

1. Create a new empty vSAN ESA cluster.

2. Download the script "restoreEsaEncryptionConfig.py" attached to this KB and upload it to the vCenter's /tmp folder (the vCenter where the hosts currently reside) using WinSCP or other file transfer tools.

3. Pull the vSAN Encryption configuration from the host to the target vSAN cluster by running the script: 

   1. Open SSH to this vCenter and cd to the /tmp folder.

   2. Run this command make the script executable: chmod +x restoreEsaEncryptionConfig.py --srcDC <source_datacenter_name> --srcCls <source_cluster_name> --dstDC <destination_datacenter_name> --dstCls <destination_cluster_name>

      Incase the cluster is under a folder object in the vCenter inventory, add the options: --srcFolder <source_folder_name> and --dstFolder <destination_folder_name>


   3. Then run this command to copy the encryption configuration from the hosts to the new vSAN ESA cluster(target): python restoreEsaEncryptionConfig.py

4. Disable HA & DRS on the current vSAN cluster (source). Prior to disabling DRS and HA, make a note of any configurations which may have been set for this, like the resource pool (this config cannot be exported, as restore cannot be performed on another cluster), admission control and so on, since this would be needed to be enabled on the target vSAN cluster when enabling HA & DRS.

5. Set "esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListUpdates" on all hosts via SSH in the cluster.

6. Select all hosts from source cluster (vSphere Client > vSAN cluster > Hosts tab), right click and select Connection > Disconnect from vCenter.

7. Once the hosts are disconnected from vCenter, drag and drop each host to the datacenter level in vCenter inventory.

8. Once all hosts are in the datacenter level, drag and drop them to the new vSAN cluster (target).

9. Select all hosts from target cluster (vSphere Client > vSAN cluster > Hosts tab), right click and select Connection > Connect to vCenter.

10. Once all hosts are connected to vCenter, set "esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates" on all hosts via SSH in the cluster.

11. Enable HA & DRS and set the required configuration for them such as resource pool, admission control and so on.

12. In vSphere Client > vSAN cluster (target) > Monitor > vSAN Skyline Health - there may be an alert for "VMware vCenter state is authoritative". Remediate it.

Additional Information

This issue is specific to vSAN ESA clusters with encryption enabled and does not apply to vSAN OSA clusters.