After upgrading vCenter to 8.0U3 vSAN Health Service triggers the alert "Cluster Configuration Consistency"

Products

• VMware vSAN

Issue/Introduction

• vSAN Cluster running version 7.x or 8.x
• vSAN Encryption is enabled
• A shallow rekey was performed on the cluster while vCenter was on version 7.x or 8.x below 8.0U3
• vCenter upgraded to 8.U3 or above the vSAN Health Service triggers the alert "Cluster Configuration Consistency" - "The host (cluster) has completed encryption configuration, but the cluster (host) has not completed."
• There is no active tasks running on vCenter with regards to encryption for the vSAN cluster.
• In vCenter health logs we see the following: 

  • /var/log/vmware/vsan-health/vmware-vsan-health-service.log reports that the vCenter encryption "changing" flag is "None" but the host encryption "changing" flag is "False":

    2024-08-05T09:55:30.769+09:00 WARNING vsan-mgmt[2679179] [VsanHealthEncUtil::_AggregateEncryptionConfigHealth opID=b4e991d2] Host: host_name has inconsistent config with cluster, field: changing, vc value: None, host value: False

  • /var/log/vmware/vsan-health/vmware-vsan-health-summary-result.log:
    2024-08-05T09:55:41.902+09:00 INFO vsan-mgmt[2679179] [VsanHealthSummaryLogUtil::PrintHealthResult opID=b4e991d2] Cluster vSAN_Cluster_name  Overall Health : red
   Group cluster health : red
      Test timedrift health : green
      Test advcfgsync health : green
      Test clomdliveness health : green
      Test resynclimit health : green
      Test vsandconfigconsistency health : green
      Test vcauthoritative health : green
      Test consistentconfig health : red
         Issues: Host  Disk  Issue  Recommendation
                 (Host-XXX, '', Host(Cluster)HasFinishedEncryptionConfigurationButCluster(Host)HasNot., Click'RemediateInconsistentConfiguration'), (Host-XXX, '', Host(Cluster)
HasFinishedEncryptionConfigurationButCluster(Host)HasNot., Click'RemediateInconsistentConfiguration'),
                 (Host-XXX, '', Host(Cluster)HasFinishedEncryptionConfigurationButCluster(Host)HasNot., Click'RemediateInconsistentConfiguration'), (Host-XXX, '', Host(Cluster)
HasFinishedEncryptionConfigurationButCluster(Host)HasNot., Click'RemediateInconsistentConfiguration'),
                 (Host-XXX, '', Host(Cluster)HasFinishedEncryptionConfigurationButCluster(Host)HasNot., Click'RemediateInconsistentConfiguration'), (Host-XXX, '', Host(Cluster)
HasFinishedEncryptionConfigurationButCluster(Host)HasNot., Click'RemediateInconsistentConfiguration'),
• Running esxcli vsan storage list|grep Encrypt on all hosts in the cluster shows all disks as encrypted
  

Environment

VMware vCenter Server 8.0U3 below 8.0U3e

VMware vSAN

 

Cause

In vCenter below 8.0U3, when a shallow rekey is performed on a vSAN encrypted cluster, vCenter records the result value as None and the host value is recorded as False, as of version 8.0U3 vCenter records the result as False to match the host value.

Since the shallow rekey was performed while vCenter was on version below 8.0U3 and then vCenter upgraded to 8.0U3 the None value carries over resulting in a false positive. 

Resolution

Clicking on the "Remediate inconsistent configuration" will NOT resolve this issue nor will upgrading the cluster to 8.0U3.

This issue is resolved in vCenter 8.0U3e build 24674346.  The vSAN Cluster Configuration Consistency alert itself is a catch-all for a number of different possible configuration issues so it's not advised to suppress the alert.

 

Additional Information

vSAN Cluster Configuration Consistency

VMware vCenter Server 8.0 Update 3e Release Notes