vSAN cluster configuration consistency alarm (blank)

Products

• VMware vSAN 8.x

Issue/Introduction

When setting up vSAN, there is an inconsistency configuration alarm occurring but has no hosts or issues detected (and is still flagged).  The cluster could have been created as OSA and then the cluster recreated via the CLI as ESA.  This could also happen when migrating ESA hosts to an OSA cluster, or vice versa. 

Example:

vSAN OSA cluster was created, but hosts were reinstalled and joined through CLI as a vSAN ESA cluster. Though the cluster is setup correctly and is healthy, due to not creating a new vSphere cluster with proper configuration set in advance, this causes a mismatch between the vCenter's and the host's understanding of the cluster storage type. 

On clicking troubleshoot, there is nothing listed in the "Issues" section.

Resolution

Create a new vSphere cluster with all the settings in place for vSAN prior. (leave DRS/HA disabled until hosts are added)

1. vSAN ESA (yes or no). 
2. VLCM set in advance (this is required for vSAN ESA, or using NSX with vSAN)
3. Data space efficiency settings must match the previous cluster settings. 
4. If encryption was in place, encryption must be configured the same as previously configured and key providers must be trusted and in sync. 
5. If custom storage policies were created, they need to be created and set for this cluster as well before moving hosts. 
6. Please also ensure 'VCLS' is in retreat mode, to clean up those VMs. 
7. Place all hosts into maintenance mode (no data migration). 

Notes:

• This is not recommended if using Horizon or any other application that auto deploys VMs as it will require re-configuration with the new cluster. 
• If NSX is in use make sure Transport Node Profiles are in use otherwise this will result in NSX getting uninstalled from the hosts. Transport Node Profiles (TNP) allows for hosts to be able to move from one cluster into another cluster within the same vCenter without the NSX getting uninstalled from the hosts.

Following these steps:

1. Re-name current cluster and add -old to end of it. 
2. Verify settings of old cluster, then create a new cluster with identical settings to the old one. (Deduplication and compression,  encryption, advanced options, etc.) 
3. SSH into each host and run the following command to avoid any potential partitions: esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListupdates
4. Disconnect each host from the old cluster. Then drag each disconnected host up into the new cluster. (Run esxcli vsan cluster get each step to make sure no partitions are happening if desired.)
5. Once all hosts moved to new cluster, right click and re-connect into the new cluster. 
6. Navigate to cluster > monitor > skyline health and all alarms should be the same as before except for the following will be triggered: vSAN Skyline Health Service - Cluster health – vCenter state is authoritative This is expected due to the advanced configuration that was set in step 3. 
7. Verify that is the only alarm being triggered and that are configurations are correct and identical between old cluster and new. 
8. SSH into hosts and reverse step 3: esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates
9. Navigate back to skyline health, click 'troubleshoot' on vCenter state is authoritative alarm. Then click 'remediate inconsistent configuration.' 

Note:  The cluster should now be in working order without the 'vSAN cluster configuration consistency' alarm being triggered.