Repeated "The metadata store is healthy" events generated every minute in vCenter Server UI

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• One or more ESXi hosts continuously generate repeated “The metadata store is healthy” events in the vCenter Server UI at frequent intervals, typically every minute.

• The issue may be observed through repeated event entries in the vCenter Server logs. Output of the journalctl -b -0 command on the vCenter Server shows entries similar to the following:

  MM DD hh:mm:ss <VC-FQDN> vpxd[3612588]: Event [503333] [1-1] [YYYY:MM:DDThh:mm:ss] [vim.event.EventEx] [info] [] [<Datacenter Name>] [503333] [The metadata store is healthy.]
MM DD hh:mm:ss <VC-FQDN> vpxd[3612588]: Event [503338] [1-1] [YYYY:MM:DDThh:mm:ss] [vim.event.EventEx] [info] [] [<Datacenter Name>] [503338] [The metadata store is healthy.]
MM DD hh:mm:ss <VC-FQDN> vpxd[3612588]: Event [503344] [1-1] [YYYY:MM:DDThh:mm:ss] [vim.event.EventEx] [info] [] [<Datacenter Name>] [503344] [The metadata store is healthy.]
MM DD hh:mm:ss <VC-FQDN> vpxd[3612588]: Event [503370] [1-1] [YYYY:MM:DDThh:mm:ss] [vim.event.EventEx] [info] [] [<Datacenter Name>] [503370] [The metadata store is healthy.]
  
  
• Logs on ESXi located at /var/run/log/hostd.log shows entries similar to the following:
  
  YYYY-MM-DDThh:mm:ss Db(167) Hostd[2522272]: [Originator@6876 sub=Hostsvc.VmkVprobSource] Read vprob : problem.metadatastore.healthy
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522272]: [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522261]: -->    key = 42,
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522261]: -->    chainId = -754630048,
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522261]: -->    createdTime = "1970-01-01T00:00:00Z",
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522261]: -->    userName = "",
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522261]: -->    host = (vim.event.HostEventArgument) {
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522261]: -->       name = "<ESXi FQDN>",
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522261]: -->       host = 'vim.HostSystem:ha-host'
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522261]: -->    },
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522261]: -->    eventTypeId = "esx.problem.metadatastore.healthy",
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522261]: -->    objectId = "ha-host",
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522261]: -->    objectType = "vim.HostSystem",
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522261]: --> }
.
.
YYYY-MM-DDThh:mm:ss In(166) Hostd[2522272]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 21637 : The metadata store is healthy.


• Additionally, the output of the following command indicates that the DKVS service is not functioning correctly on the ESXi host. The "available": false status indicates that DKVS is not running properly on the host.
  
  [root@esxi:~] /usr/lib/vmware/clusterAgent/bin/clusterAdmin cluster status
{
   "state": "hosted",
   "cluster_id": "<Cluster UUID>:domain-c8",
   "is_in_alarm": true,
   "alarm_cause": "Timeout",
   "is_in_cluster": true,
   "members": {
      "available": false
   },
}

Note: In some environments, the cluster_id field may appear empty, which also indicates that DKVS is not operational on the ESXi host.

Environment

vCenter Server 8.x
vSphere ESXi 8.x

Cause

This issue occurs when the DKVS (Distributed Key-Value Store) cluster enters an unhealthy or error state. As a result, ESXi host continuously generates metadata store health events, which are then forwarded to vCenter Server and displayed repeatedly in the vSphere UI.

Resolution

1. Review and implement the workaround steps documented in the following KB article to disable DKVS service: ESXi hosts constantly sending DNS queries to the DNS server
   
   Note:
   The DKVS service is used during vCenter restore from backup to provide more up-to-date recovery. If you take regular vCenter backups, disabling DKVS is generally safe. If DKVS is disabled and the vCenter backup differs from the current host inventory configuration (host membership, credentials, DVS state), vCenter recovery may cause host disconnects that require manual reconnection.
   
   
2. If the workaround of disabling DKVS service is not feasible, contact Broadcom Technical Support for further assistance.

Additional Information

Example output when DKVS is running in healthy state:

[root@esxi:~] /usr/lib/vmware/clusterAgent/bin/clusterAdmin cluster status
{
   "state": "hosted",
   "cluster_id": "######",
   "is_in_alarm": false,
   "alarm_cause": "",
   "is_in_cluster": true,
   "members": {
      "available": true
   },
   "namespaces": [
      {
         "name": "root",
         "up_to_date": true,
         "members": [
            {
               "peer_address": "######",
               "api_address": "######",
               "reachable": true,
               "primary": "yes",
               "learner": false
            },
            {
               "peer_address": "######",
               "api_address": "######",
               "reachable": true,
               "primary": "no",
               "learner": false
            },
            {
               "peer_address": "######",
               "api_address": "######",
               "reachable": true,
               "primary": "no",
               "learner": false
            }
         ]
      }
   ]
}