vSAN Health Alarm 'vCenter and all hosts are connected to Key Management Servers' intermittently flapping from green to yellow and then back to green
search cancel

vSAN Health Alarm 'vCenter and all hosts are connected to Key Management Servers' intermittently flapping from green to yellow and then back to green

book

Article ID: 439658

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:

  • The vSAN Skyline Health check "vCenter and all hosts are connected to Key Management Servers" intermittently changes from Green to Yellow and then back to Green. Rarely it would also intermittently switch to red status.

  • A vCenter alarm is detected: alarm.vsan.health.test.encryption.kmsconnection:

vCenter alarm detected requiring attention. Alarm details: alarm.vsan.health.test.encryption.kmsconnection - Event: Checks if VMware vCenter or any hosts are not connected to Key Management Servers (330357654) Summary: vSAN Health Test 'vCenter and all hosts are connected to Key Management Servers' status changed from 'green' to 'red' Date: 05/02/2026 12:12:00 AM User name: vSAN Health Resource pool: SG4-######-######### Data center: Singapore Arguments: eventTypeId = vsan.health.test.encryption.kmsconnection.event objectId = domain-c7729 objectName = SG4-######-######### severity = error prestatus = green curstatus = red

  • The vSAN cluster remains healthy and functional.
  • The following error appears in vsan-health-service.log or vsanmgmt.log: Exception: Function CheckKmsStatusForDefaultView is timeout.

On vCenter under /var/log/vmware/vsan-health/vsan-health-service.log, the below error shows up:

2026-05-04T22:46:05.802Z WARNING vsan-mgmt[2675953] [VsanHealthEncUtil::_AggregateEncryptionConfigHealth opID=noOpId] Host: KMSserver.com encryption health error: (vim.fault.VsanFault) {
  msg = '',
  faultMessage = (vmodl.LocalizableMessage) [
    (vmodl.LocalizableMessage) {
      key = 'com.vmware.vsan.health.msg.host.runtime.error',
      message = 'Function CheckKmsStatusForDefaultView is timeout.'

On vSAN node under /var/log/vmware/vsanmgmt.log the below timeout error shows up:

2026-05-03T11:42:20.354Z Er(11) vsand[2099488]: [opID=a8a71796-fcab VsanHealthEncUtil::GenerateEncryptionHealthSummary] Failed to get host encryption info
2026-05-03T11:42:20.354Z Er(11)[+] vsand[2099488]: Traceback (most recent call last):
2026-05-03T11:42:20.354Z Er(11)[+] vsand[2099488]:   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthEncUtil.py", line 477, in GenerateEncryptionHealthSummary
2026-05-03T11:42:20.354Z Er(11)[+] vsand[2099488]:   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthEncUtil.py", line 246, in _CheckKmsStatus
2026-05-03T11:42:20.354Z Er(11)[+] vsand[2099488]:   File "/usr/lib/vmware/vsan/perfsvc/VsanVcObjectHelper.py", line 643, in wrapper
2026-05-03T11:42:20.354Z Er(11)[+] vsand[2099488]: Exception: Function CheckKmsStatusForDefaultView is timeout.

  • Doing a ping test from the ESXi host to the KMS server taken long time to receive a response (approximately 200ms).

Environment

VMware vSAN 7.x, 8.x

Cause

  • The issue is caused by the ESXi host failing to complete the KMS health check within a mandatory 20-second timeout window.
  • Analysis of the vsanmgmt.log indicates that when the host attempts to fetch encryption information, the function CheckKmsStatusForDefaultView is called at each host to examine the kms server health, and the error message 'Function CheckKmsStatusForDefaultView is timeout' indicates that host cannot finish checking all kms server health status within the timeout(20s).

2026-05-03T11:42:00.350Z In(14) vsand[2099488]: [opID=a8a71796-fcab VsanVcObjectHelper::wrapper] Ready to get single executor result for the key ['CheckKmsStatusForDefaultView', 'clusterId:id:KMS_Cluster;info:KMSserveraddress;name:KMSserver;port:5696,clusterId:id:KMS_Cluster;info:address:KMSserver:blk... in timeout 20
.
2026-05-03T11:42:20.353Z In(14) vsand[2099488]: [opID=a8a71796-fcab VsanVcObjectHelper::wrapper] Fetch from cache for CheckKmsStatusForDefaultView
2026-05-03T11:42:20.354Z In(14) vsand[2099488]: [opID=a8a71796-fcab VsanVcObjectHelper::getResultFromCache] The cache is timeout for CheckKmsStatusForDefaultView, time: 1777804716.748717, value: [(vim.host.VsanKmsHealth) {
2026-05-03T11:42:20.354Z In(14)[+] vsand[2099488]:   serverName = 'KMSserver1',
2026-05-03T11:42:20.354Z In(14)[+] vsand[2099488]:   health = 'green'
2026-05-03T11:42:20.354Z In(14)[+] vsand[2099488]: }, (vim.host.VsanKmsHealth) {
2026-05-03T11:42:20.354Z In(14)[+] vsand[2099488]:   serverName = 'KMSserver2',
2026-05-03T11:42:20.354Z In(14)[+] vsand[2099488]:   health = 'green'
2026-05-03T11:42:20.354Z In(14)[+] vsand[2099488]: }, (vim.host.VsanKmsHealth) {
2026-05-03T11:42:20.354Z In(14)[+] vsand[2099488]:   serverName = 'KMSserver3',
2026-05-03T11:42:20.354Z In(14)[+] vsand[2099488]:   health = 'green'
2026-05-03T11:42:20.354Z In(14)[+] vsand[2099488]: }, (vim.host.VsanKmsHealth) {
2026-05-03T11:42:20.354Z In(14)[+] vsand[2099488]:   serverName = 'KMSserver4',
2026-05-03T11:42:20.354Z In(14)[+] vsand[2099488]:   health = 'green'
2026-05-03T11:42:20.354Z In(14)[+] vsand[2099488]: }]
2026-05-03T11:42:20.354Z Er(11) vsand[2099488]: [opID=a8a71796-fcab VsanHealthEncUtil::GenerateEncryptionHealthSummary] Failed to get host encryption info
2026-05-03T11:42:20.354Z Er(11)[+] vsand[2099488]: Traceback (most recent call last):
2026-05-03T11:42:20.354Z Er(11)[+] vsand[2099488]:   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthEncUtil.py", line 477, in GenerateEncryptionHealthSummary
2026-05-03T11:42:20.354Z Er(11)[+] vsand[2099488]:   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthEncUtil.py", line 246, in _CheckKmsStatus
2026-05-03T11:42:20.354Z Er(11)[+] vsand[2099488]:   File "/usr/lib/vmware/vsan/perfsvc/VsanVcObjectHelper.py", line 643, in wrapper
2026-05-03T11:42:20.354Z Er(11)[+] vsand[2099488]: Exception: Function CheckKmsStatusForDefaultView is timeout.

  • The task times out after exactly 20 seconds if a response is not received from the KMS server. This delay is typically attributed to high network latency or processing delays on the KMS provider's side.
  • In observed cases, average network latencies of 200ms or higher have contributed to exceeding this timeout threshold.

Resolution

To resolve this issue, you must reduce the response time between the ESXi hosts and the KMS servers:

  1. Test Network Latency: From the ESXi host command line, perform a ping test to the KMS server to identify high response times. ping <KMS_IP_Address>
  2. Verify Port Connectivity: Confirm that bidirectional communication is permitted on TCP Port 5696. Use the following command on the ESXi host to verify the port is open: nc -zv <KMS_IP_Address> 5696
  3. Run Latency Script: Execute the Python latency test script to measure the specific time cost for the CheckKmsStatus function. (Kindly open a Broadcom Support ticket to help running the test)
  4. Engage Vendor Support: If high latency is confirmed, coordinate with your network or KMS vendor teams to optimize the connection.
Powered by Wolken Software