Unable to query vSAN health information due to KMS/VECS permission issues

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Unable to query vSAN health information. Check vSphere Client logs for details.
• After the upgrade of the vCenter server, Skyline Health is unable to update the status.

/var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log

=======================================================
Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: https://<VC-FQDN / IP:443/vsanHealth invocation failed with "java.net.SocketTimeoutException: Read timed out"
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:261)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:311)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:57)
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:51)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:227)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:114)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195)
        at com.sun.proxy.$Proxy1028.queryClusterHealthSummary(Unknown Source)
        at com.vmware.vsphere.client.vsan.health.VsanHealthPropertyProvider.getClusterHealthSummary(VsanHealthPropertyProvider.java:293)
        at com.vmware.vsphere.client.vsan.health.VsanHealthPropertyProvider.getVsanHealthSummaryFromCache(VsanHealthPropertyProvider.java:273)
        ... 167 common frames omitted

/var/log/vmware/vsan-health/vmware-vsan-health-service.log

===========================================================
YYYY-MM-DDTHH:MM:SSZ ERROR vsan-mgmt[09513] [VsanClusterPrototypeImpl::RemediateCluster opID=####-###-##########-sq3:j#-###] RemediateCluster failed: unidentifiable C++ exception
Traceback (most recent call last):
  File "/build/mts/release/bora-17327517/bora/build/vcenter/release/pkg/vsan-health/stage/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanClusterPrototypeImpl.py", line 2997, in RemediateCluster
  File "/build/mts/release/bora-17327517/bora/build/vcenter/release/pkg/vsan-health/stage/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanClusterPrototypeImpl.py", line 3156, in _RemediateCluster
  File "/build/mts/release/bora-17327517/bora/build/vcenter/release/pkg/vsanmgmt/stage/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanVcEncryption.py", line 63, in __enter__
  File "/usr/lib/vmware/site-packages/identity/vmkeystore.py", line 50, in aliases
    self.enum_context = self._client.BeginEnumAliases(self._store_context, 10)
RuntimeError: unidentifiable C++ exception

/var/log/vmware/vmafdd/vmafdd.log

=======================================
YYYY-MM-DDTHH:MM:SSZ [vmafdd][ERROR] ERROR! VecsIpcBeginEnumEntries failed. Exiting with error : [5]

Environment

VMware vCenter Server 7.0.x

Cause

When the environment is configured with KMS, the permission needs to be reset for all users.

Resolution

This issue is resolved in VMware vCenter 7.0 Update 2. 

Workaround:
#/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd --user KMS_ENCRYPTION --grant read
# service-control --restart vmware-vsan-health