vSAN cluster configuration consistency triggered to yellow once, but automatically restored to green

Products

• VMware vSAN

Issue/Introduction

vSAN cluster configuration consistency triggered to yellow once, but automatically restored to green in the next health check.

From vmware-vsan-health-service.log, .vsan.stats shows mismatch for spbmComplianceResult.

YYYY-MM-DDTHH:MM:SS.MS+HH:MM INFO vsan-mgmt[17290] [VsanVcPerformanceManagerImpl::QueryClusterHealth opID=noOpI#] QueryClusterHealth objInfo: (vim.cluster.VsanObjectInformation) {
  directoryName = '.vsan.stats',
  vsanObjectUuid = '########-####-####-####-############',
  vsanHealth = 'healthy',
:::
  spbmProfileUuid = '########-####-####-####-############',
  spbmProfileGenerationId = '0',
  spbmComplianceResult = (vim.cluster.StorageComplianceResult) {
    checkTime = YYYY-MM-DDTHH:MM:SS.MSZ,
    profile = '########-####-####-####-############',
    objectUUID = '########-####-####-####-############',
    complianceStatus = 'compliant',
    mismatch = true,         ###<- !!!
    operationalStatus = (vim.cluster.StorageOperationalStatus) {
      healthy = true
    },
    objPolicyGenerationId = '0'
  }
}

Also, the log below indicates connection failure to sps service.

YYYY-MM-DDTHH:MM:SS.MS+HH:MM ERROR vsan-mgmt[29331] [VsanVcObjectHelper::isMismatch opID=noOpI#] Cannot get profile for id ########-####-####-####-############, error is (vmodl.RuntimeFault) {
  msg = 'Received SOAP response fault from [<, /pbm/sdk>]: <MASKED DOMAIN\USER> persistence service not available.',
  faultCause = (vmodl.RuntimeFault) {
    msg = 'KV persistence service not available.',
    faultCause = (vmodl.RuntimeFault) {
      msg = 'Unexpected status code: 503',
      faultMessage = (vmodl.LocalizableMessage) [
        (vmodl.LocalizableMessage) {
          key = 'com.vmware.pbm.pbmFault.locale',
          arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
              key = 'summary',
              value = 'Unexpected status code: 503'
            }
          ]
        }
      ]
    },
    faultMessage = (vmodl.LocalizableMessage) [
      (vmodl.LocalizableMessage) {
        key = 'com.vmware.pbm.pbmFault.locale',
        arg = (vmodl.KeyAnyValue) [
          (vmodl.KeyAnyValue) {
            key = 'summary',
            value = 'KV persistence service not available.'
          }
        ]
      }
    ]
  },
  faultMessage = (vmodl.LocalizableMessage) [
    (vmodl.LocalizableMessage) {
      key = 'com.vmware.pbm.pbmFault.locale',
      arg = (vmodl.KeyAnyValue) [
        (vmodl.KeyAnyValue) {
          key = 'summary',
          value = 'KV persistence service not available.'
        }
      ]
    }
  ]
}

Environment

VMware vSAN

Cause

There is a known issue that sps service becomes unresponsive for a short time. When vsan-health service failed to get storage profile information that is used for .vsan.stats object, its spbmComplianceResult becomes mismatch.

As a result, vSAN configuration consistency alarm becomes yellow.

Resolution

Currently, there is no workaround of this issue. Since it only triggers vSAN configuration consistency once and doesn't affect to vSAN functionality, it can be safely ignored.