"vCenter and all hosts are connected to Key Management Servers" warning is shown in Skyline Health when machine user privileges are not sufficient

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• In Skyline Health, for a given vSAN cluster, you see the following warning:

vCenter and all hosts are connected to Key Management Servers

• When reviewing further details on this warning, you see the following for each host in the cluster:

Unable to fetch key provider details on host.unknown list provider error, please check health logs

• You have otherwise determined that the KMS servers should already be reachable by vCenter Server, and the respective ESXi hosts, and there do not appear to be any serious functionality or availability issues with the affected cluster or clusters.

Cause

• This issue can be caused by absent VSPHERE.LOCAL\Administrators group privilege propagation down the cluster and host objects.
  
  
• At least upon initial inspection, the Skyline Health warning is shown in the /var/log/vmware/vsan-health/vmware-vsan-health-service.log files in a manner similar to the following:

<YYYY-MM-DD>T<time>ERROR vsan-mgmt[11928] [VsanHealthEncUtil::GenerateClusterEncryptionHealthSummary opID=noOpId] host hostname.domain.com, encryptionIssues: (vim.host.VsanEncryptionIssue) [], encryptionInfo (vim.vsan.host.EncryptionInfo) {

 enabled = true,

 kekId = '67b1c4b71d5f21aaa7e885ef5d538007c091db33:b04b5168-ecce-21ec-acda-100c29d08071',

 hostKeyId = 'kmx:36 b04b57c6-ecce-21ec-acda-100c29d08071 BSEAAgEAYUx7/Eh/Qxatp236B9mG7QEIAAwAEAAgAAQAQUVTLTI1NgDgcRPvqMvNb/wk9zrA/iui7xdXT+72mByoqEAeHPQdBixnL5thHPp9vHb665IT6oZ9GmXzBadRIVnxRPWghgEA',

 kmipServers = (vim.encryption.KmipServerSpec) [

  (vim.encryption.KmipServerSpec) {

   clusterId = (vim.encryption.KeyProviderId) {

    id = 'vcenter-server-name-KMS'

   },

   info = (vim.encryption.KmipServerInfo) {

    name = 'NativeKeyProvider',

    address = 'nkp:kmx',

    port = 0

   }

  }

 ],

 dekGenerationId = 1,

 changing = false,

 eraseDisksBeforeUse = false

},            error (vim.fault.VsanFault) {

 faultMessage = (vmodl.LocalizableMessage) [

  (vmodl.LocalizableMessage) {

   key = 'com.vmware.vsan.health.msg.list.kmxa.provider.error',

   message = 'get provider info error, please check the health logs'

  }

 ]

}, kmsHealth (vim.host.VsanKmsHealth) [

 (vim.host.VsanKmsHealth) {

  serverName = 'NativeKeyProvider',

  health = 'red',

  error = (vim.fault.VsanFault) {

   faultMessage = (vmodl.LocalizableMessage) [

    (vmodl.LocalizableMessage) {

     key = 'com.vmware.vsan.health.msg.list.kmxa.provider.error',

     message = 'unknown kmx provider error'

    }

   ]

  }

 }

], aesniEnabled True

 

• More details can be seen on individual ESXi servers, particularly in the /var/run/log/vsanmgmt.log files, once you locate a corresponding timestamp and event:

<YYYY-MM-DD>T<time>ERROR vsan-mgmt[12397] [VsanHealthEncUtil::GenerateEncryptionHealthSummaryForKmx opID=noOpId] Error when GetVpxdHostProviderInfo: hostname.domains.com

Traceback (most recent call last):

File "bora/vsan/health/esx/pyMo/VsanHealthEncUtil.py", line 393, in GenerateEncryptionHealthSummaryForKmx

File "bora/vsan/clusterconfig/vpxd/pyMoVsan/VsanVcEncryption.py", line 429, in GetVpxdHostProviderInfo

File "/usr/lib/vmware-vpx/vsan-health/vapi/vapi_vcenter_client-1.0-py2.7.egg/com/vmware/vcenter/crypto_manager/hosts/kms_client.py", line 406, in get

'provider': provider,

File "/usr/lib/vmware-vpx/vsan-health/vapi/vapi_runtime-2.100.0.egg/vmware/vapi/bindings/stub.py", line 345, in _invoke

return self._api_interface.native_invoke(ctx, _method_name, kwargs)

File "/usr/lib/vmware-vpx/vsan-health/vapi/vapi_runtime-2.100.0.egg/vmware/vapi/bindings/stub.py", line 298, in native_invoke

self._rest_converter_mode)

com.vmware.vapi.std.errors_client.Unauthorized: {messages : [LocalizableMessage(id='vapi.authz.error.no.privs', default_message='The following (object: host-40:ddee9d5e-66b5-478e-a19c-585efb2ee7d6 privileges: Cryptographer.ReadKeyServersInfo) privileges are insufficient to user', args=['object: >host-40:edee9d5e-76b5-578e-b19c-685efb2ee7d6 privileges: Cryptographer.ReadKeyServersInfo'], params=None, localized=None)], data : None, error_type : UNAUTHORIZED}

<YYYY-MM-DD>T<time>INFO vsan-mgmt[12397] [VsanHealthEncUtil::GenerateEncryptionHealthSummaryForKmx opID=noOpId] host: hostname.domain.com kmx health error: unknown kmx provider error

 

• The error above indicates that there are permissions or privilege constraints for Cryptographer.ReadKeyServersInfo, which in the vSphere Client, falls under the Cryptographic operations -> Read KMS information privilege, specifically.

Resolution

• To resolve this issue, ensure that the VSPHERE.LOCAL\Administrators group has sufficient permissions at the Cluster and Host object levels.
  
  
• In the vSphere Client's default Hosts & Clusters view, select the top-level vCenter Server object, and then the Permissions tab.
  Review the VSPHERE.LOCAL\Administrators group in the list and ensure that it has the Administrator role (or one with similar permissions) is shown.
  The Defined In column should show This object and its children. If it does not, edit the Administrators group, and assign the Administrator role, select the Propagate to Children checkbox, then click OK.
  Note: If required, you can instead apply this at a lower branch object, such as at the Datacenter, Folder, or Cluster object level. However, by default, this is expected to be propagated from the top-level vCenter Server level, all the way down to the individual clusters and hosts.
  
  
• Upon re-testing Skyline Health for the affected cluster(s), you should see the previously failing test for "vCenter and all hosts are connected to Key Management Servers" now pass. If not, then you may have a legitimate connectivity or communication issue with the KMS server and will need to troubleshoot further.