Virtual machines experience performance degradation or become unresponsive when the CMMDS leader ESXi host is rebooted
search cancel

Virtual machines experience performance degradation or become unresponsive when the CMMDS leader ESXi host is rebooted

book

Article ID: 407826

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:

Post reboot of the CMMDS Leader of the vSAN cluster (Server cluster CMMDS Leader in case of vSAN HCI Mesh):

  • VMs may be slow or unresponsive 
  • In clustered environments like Oracle RAC, member nodes may get rebooted if the I/O latency becomes more than the interconnect latency threshold between the cluster nodes
  • LUN may show 100% utilization with no active read/write operations.
  • The cluster may enter a vSAN network partition state during the leader node reboot.
  • The vSAN iSCSI Target Service (daemon process vitd) may run out of available threadpool resources.
  • VMs may experience connection errors (vSAN iSCSI)

mmm dd hh:mm:ss vm_name kernel: [1761594.974483] connection5:0: detected conn error (1020)
mmm dd hh:mm:ss vm_name kernel: [1761594.974482] connection6:0: detected conn error (1020)

  • In vSAN HCI Mesh environments:
    1. Leader may lose connectivity with the other server nodes but still maintain connectivity with the client nodes
    2. DOM clients may lose connectivity to corresponding DOM owners (DOM Owners may be marked unhealthy in the client cluster):

 

Environment

  • VMware vSAN 8.0 P05 or lower

Cause

A known CMMDS issue where a leader host undergoing reboot or shutdown continued transmitting leader heartbeats, even though it was no longer able to receive traffic.

Because the other cluster nodes were still receiving these outgoing heartbeats, they continued to follow the rebooting leader instead of failing over to the backup. As a result, a clean leadership transition was prevented, and a full cluster partition was triggered until the leader’s networking stack was fully stopped.

 

Logs Validation:

The issue can be confirmed through the following log observations:

1. Initial Leader State 
  • Leader of vSAN cluster before MM entry: (/var/run/log/vmkernel.log)
2025-08-02T01:42:55.501Z In(182) vmkernel: cpu73:2098937)CMMDS: LeaderBuildHeartbeatMessage:2120: 52cfc3d8-####-####-ebd1-#########: [318070950]:Current membership uuid 33c41967-####-####-23a9-######### has 14 members
2025-08-02T01:42:55.501Z In(182) vmkernel: cpu73:2098937)CMMDS: LeaderBuildHeartbeatMessage:2131: 52cfc3d8-####-####-ebd1-#########: [318070950]:Member[0]:6718c2b8-####-####-fbf3-#########(leader)
2025-08-02T01:42:55.501Z In(182) vmkernel: cpu73:2098937)CMMDS: LeaderBuildHeartbeatMessage:2126: 52cfc3d8-####-####-ebd1-#########: [318070950]:Member[1]:6719af12-####-####-ff6c-#########(backup)
2. Leader Enters Maintenance Mode: (/var/run/log/vobd.log)
2025-08-02T02:26:35.617Z In(14) vobd[2098025]:  [UserLevelCorrelator] 24423759940313us: [esx.audit.maintenancemode.entered] The host has entered maintenance mode.
3. Leader Reboot Initiated: (/var/run/log/vmksummary.log)
2025-08-02T02:26:36.568Z No(13) bootstop[245573871]: Host is rebooting
4. Backup Node Becomes Leader
  • Takes over leadership: (/var/run/log/vmkernel.log)
2025-08-02T02:26:54.480Z In(182) vmkernel: cpu86:2099008)CMMDSNet: CMMDSNet_SetLeader:1315: 52cfc3d8-####-####-ebd1-#########: Updating leader node: old=6718c2b8-####-####-fbf3-######### new=none
2025-08-02T02:26:54.480Z In(182) vmkernel: cpu86:2099008)CMMDSNet: CMMDSNet_SetLeader:1315: 52cfc3d8-####-####-ebd1-#########: Updating leader node: old=none new=6719af12-####-####-ff6c-#########
2025-08-02T02:26:54.480Z In(182) vmkernel: cpu86:2099008)CMMDS: CMMDSLogStateTransition:1824: 52cfc3d8-####-####-ebd1-#########: Transitioning(6719af12-####-####-ff6c-#########) from Backup to Leader: (Reason: Backup is taking over the cluster leader)
5. Agent Node Behavior
  • Instead of failing over to a new leader, it drops to discovery and Rejoin states: (/var/run/log/vmkernel.log) - This happens when the agents are not aware of the new leader:
2025-08-02T02:26:59.486Z In(182) vmkernel: cpu50:2099008)CMMDS: CMMDSLogStateTransition:1824: 52cfc3d8-####-####-ebd1-#########: Transitioning(6719b151-####-####-1fc1-#########) from Agent to Discovery: (Reason: Failed to receive from node)
2025-08-02T02:27:00.064Z In(182) vmkernel: cpu57:2099008)CMMDS: CMMDSLogStateTransition:1824: 52cfc3d8-####-####-ebd1-#########: Transitioning(6719b151-####-####-1fc1-#########) from Discovery to Rejoin: (Reason: Found a leader node)
2025-08-02T02:27:00.329Z In(182) vmkernel: cpu57:2099008)CMMDS: CMMDSLogStateTransition:1824: 52cfc3d8-####-####-ebd1-#########: Transitioning(6719b151-####-####-1fc1-#########) from Rejoin to Discovery: (Reason: Failed to receive from node)
2025-08-02T02:27:01.814Z In(182) vmkernel: cpu57:2099008)CMMDS: CMMDSLogStateTransition:1824: 52cfc3d8-####-####-ebd1-#########: Transitioning(6719b151-####-####-1fc1-#########) from Discovery to Rejoin: (Reason: Found a leader node)
2025-08-02T02:27:03.610Z In(182) vmkernel: cpu57:2099008)CMMDS: CMMDSLogStateTransition:1824: 52cfc3d8-####-####-ebd1-#########: Transitioning(6719b151-####-####-1fc1-#########) from Rejoin to Agent: (Reason: The local node has finished rejoining)
  • In some conditions, it may directly transition to a Rejoin state to the new leader instead of failing over - This happens when agents are aware of the new leader but the old leader still considers itself to be the CMMDS master:

2026-03-08T01:54:42.290Z In(182) vmkernel: cpu37:2099050)CMMDSNet: CMMDSNet_SetLeader:1330: 85fb70a4-3b0d-48e7-b4d2-#######: Updating leader node: old=6823bede-7656-3c25-73c8-######### new=none
2026-03-08T01:54:42.290Z In(182) vmkernel: cpu37:2099050)CMMDS: CMMDSLogStateTransition:1838: 85fb70a4-3b0d-48e7-b4d2-########: Transitioning(63eb609a-7fc4-b602-fdbe-########) from Client to Rejoin: (Reason: Found a leader node)

  • Agent Node will detect the loss of its leader host and terminates the active RDT association with it: (/var/run/log/vmkernel.log)
2025-08-02T02:26:59.486Z In(182) vmkernel: cpu59:2099008)CMMDS: CMMDSStateMachineReceiveLoop:1640: 52cfc3d8-####-####-ebd1-#########: Error receiving from 6718c2b8-####-####-fbf3-#########: Failure 2025-08-02T02:26:59
2025-08-02T02:26:59.486Z In(182) vmkernel: cpu59:2099008)CMMDS: CMMDSStateDestroyNode:708: 52cfc3d8-####-####-ebd1-#########: Destroying node 6718c2b8-####-####-fbf3-#########: Failed to receive from node
2025-08-02T02:26:59.486Z In(182) vmkernel: cpu59:2099008)CMMDS: AgentDestroyNode:1660: 52cfc3d8-####-####-ebd1-#########: Lost leader node (6718c2b8-####-####-fbf3-#########), can't handle that and will transition to discovery
2025-08-02T02:26:59.486Z In(182) vmkernel: cpu59:2099008)CMMDSNet: CMMDSNet_SetLeader:1315: 52cfc3d8-####-####-ebd1-1c05 #########: Updating leader node: old=6718c2b8-####-####-fbf3-######### new=none
2025-08-02T02:26:59.486Z In(182) vmkernel: cpu50:2099008)CMMDS: CMMDSLogStateTransition:1824: 52cfc3d8-####-####-ebd1-#########: Transitioning(6719b151-####-####-1fc1-#########) from Agent to Discovery: (Reason: Failed to receive from node)
  • RDT termination may also be noticed in the vSAN Default traces:

2026-03-08T01:54:42.289257 [174889790] [cpu63] [CLIENT] DOMTraceObjectServerAssocTerminateCb:2115: {'obj': 0x45bb15c0a3c0, 'objUuid': '0bbaa369-06f5-d456-3a52-#######', 'assoc': 0x4336451ae500, 'leafState': 'invalid', 'isCleaningUp': False}
2026-03-08T01:54:42.455407 [174890035] [cpu63] [CLIENT] DOMTraceObjectReconnectOpToObj:1995: {'obj': 0x45bb15c0a3c0, 'objUuid': '0bbaa369-06f5-d456-3a52-#######', 'isInitTSInPast': True, 'numReconnect': 1}

  • The Agent Nodes, may continue to receive heartbeats from the old leader until it is transitions to Rejoin state: (/var/run/log/vmkernel.log):
2025-08-02T02:26:59.488Z In(182) vmkernel: cpu50:2099008)CMMDS: CMMDSClusterDestroyNodeImpl:262: Destroying node 6719af12-####-####-ff6c-######### from the cluster db. Last HB received from node - 24349239336243137
6. VM Object Impact
  • DOM loses liveness for impacted VM: (/var/run/log/vmkernel.log)
2025-08-02T02:26:59.481Z In(182) vmkernel: cpu13:2099082)DOM: DOMOwner_SetLivenessState:10887: Object 16043e67-####-####-1a51-######### lost liveness [0x45bb80a3f840]


Resolution

  1. This issue is addressed in VMware ESXi 8.0.3 build-24859861 ( ESXi 8.0 P06) and in ESXi 9.0.

    • It is recommended to upgrade the affected ESXi hosts to one of these fixed versions to prevent recurrence.

  2. Workaround:
    • Manually abdicate the vSAN cluster leader role before rebooting:

      Abdicate the leader

      # vsish -e set /vmkModules/cmmds/forceTransition abdicateLeader

      OR
    • Temporarily untag the vSAN vmknic before rebooting:

      • Remove the vSAN traffic tag from the vmknic prior to host reboot.

         To untag vSAN traffic run the below command:

                 #  esxcli network ip interface tag remove -i vmkx -t vSAN

         

      • Re-apply the vSAN tag after the host has successfully rebooted.

         To re-tag after upgrade and reboot run the below command:

               #  esxcli network ip interface tag add -i vmkx -t vSAN

         

Powered by Wolken Software