vSAN Objects Inaccessible due to MTU Mismatch causing Cluster Partition
search cancel

vSAN Objects Inaccessible due to MTU Mismatch causing Cluster Partition

book

Article ID: 444421

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms: 

  • Multiple Virtual Machines (VMs) and the vCenter Server are currently inaccessible.

  • All cluster hosts report as partitioned.

  • All hosts within the cluster experience a network partition

Environment

VMware vSAN 8.x (OSA)

Cause

  • The cluster partition was caused by an MTU mismatch on the physical network path for master node , resulting in a partial network failure on the CMMDS Master node.
  • vSAN relies on Cluster Monitoring, Membership, and Directory Services (CMMDS) for cluster health and metadata. Standard CMMDS heartbeats utilize small packets to maintain leadership.
  • Using a lower MTU allows small heartbeat packets to travel across the misconfigured physical network. This causes the surviving nodes to detect a healthy Master, which prevents a clean failover
  • vSAN metadata and directory synchronization require large packet payloads. When attempting to transmit these updates using Jumbo Frames (MTU 8972/9000), the physical switch silently dropped the traffic due to the MTU restriction, breaking cluster communication and causing the partition.

Cause Validation :

  • The diagnosis is supported by vmkping test results and ESXi log evidence from the affected hosts during the partition event, confirming that small payloads passed while large payloads dropped

    Ping Test Validation

    Ping test with 400 MTU from the affected host to other hosts: Successful

    [root@#####-exmg03 :~ ] vmkping -I vmk2 ###### -d -5 400
    PING ####(1####3): 400 data bytes
    408 bytes from #####: icmp seg=0 ttl=64 time=39.713 ms
    408 bytes from ####: icmp seg=1 ttl=64 time=36.979 ms

    -- ###### ping statistics ---
    3 packets transmitted, 3 packets received, 0% packet loss
    round-trip min/avg/max = 33.979/36.890/39.713 ms

    [root@####exmg03 :~ ] vmkping -I vmk2 ##### -d -5 472
    PING ##### (#####): 472 data bytes
    480 bytes from #####: icmp seg=1 ttl=64 time=401.090 ms
    480 bytes from #######: icmp seg=2 ttl=64 time=874.839 ms

    --- 1###### ping statistics ---
    3 packets transmitted, 2 packets received, 33.3333% packet loss
    round-trip min/avg/max = 401.090/637.965/874.839 ms

    Ping test with >500 MTU from the affected host to other hosts: Failed (100% packet loss)

    [root@###exmg03 :~ ] vmkping -I vmk0 ### -d -5 1472
    PING ##### (#######): 1472 data bytes

    -- ###### ping statistics --
    3 packets transmitted, 0 packets received, 100% packet loss


    [root@h-img-pr-exmg03 :~ ] vmkping -I vmk0 1####2 -d -5 1472
    PING ##### (#####): 1472 data bytes

    --##### ping statistics --
    3 packets transmitted, 0 packets received, 100% packet loss

    [root@h-img-pr-exmg03 :~ ] vmkping -I vmk0 ##### -d -s 1400
    PING #### (######): 1400 data bytes
    ---##### ping statistics --
    3 packets transmitted, 0 packets received, 100% packet loss

    Without using MTU ping succeeds 

    [root@h-img-pr-exmg03 :~ ] vmkping -I vmk0 ######
    PING ####(#####): 56 data bytes
    64 bytes from #####: icmp seg=0 ttl=62 time=6.611 ms
    64 bytes from 1####: icmp.seg=1 ttl=62 time=238.266 u5
    64 bytes from 1##### icmp seg=2 ttl=62 time=293.571 ms

  • Verification of (less /var/run/log/vsansystem.log) Cluster partition started after stopped receiving metadata information from master node 

    2026-05-27T16:11:29.620Z In(166) vsansystem[2104044]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSMembershipUpdate-fe89] Complete, nodeCount: 7, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
    2026-05-27T16:11:40.033Z In(166) vsansystem[2104029]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSMembershipUpdate-fe95] Complete, nodeCount: 6, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
    2026-05-27T16:12:19.765Z In(166) vsansystem[2104012]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSMembershipUpdate-feca] Complete, nodeCount: 4, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
    2026-05-27T16:12:20.260Z In(166) vsansystem[2108207]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSMembershipUpdate-fed5] Complete, nodeCount: 4, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
    2026-05-27T16:12:41.019Z In(166) vsansystem[2104044]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSMembershipUpdate-fed7] Complete, nodeCount: 3, runtime info: (vim.vsan.host.VsanRuntimeInfo) {

     

  • Verification of vmkernel.log of master host (affected host) (less /var/run/log/vmekernel.log) confirms the Backup Node became unresponsive and was destroyed in the cluster database due to the inability to receive large CMMDS updates, despite some heartbeat sequences still attempting to filter through.

    2026-05-27T16:00:27.755Z In(182) vmkernel: cpu43:2100196)CMMDS: LeaderBuildHeartbeatMessage:2441:  #########################c [24230424]:Member[0]:######(leader)
    2026-05-27T16:00:27.755Z In(182) vmkernel: cpu43:2100196)CMMDS: LeaderBuildHeartbeatMessage:2436: #######################: [24230424]:Member[1]:###########(backup)
    2026-05-27T16:00:27.755Z In(182) vmkernel: cpu43:2100196)CMMDS: LeaderBuildHeartbeatMessage:2445: #######################: [24230424]:Member[2]:###############(agent)
    2026-05-27T16:00:27.755Z In(182) vmkernel: cpu43:2100196)CMMDS: LeaderBuildHeartbeatMessage:2445: ##################:      [24230424]:Member[3]:##########0(agent)
    2026-05-27T16:00:27.755Z In(182) vmkernel: cpu43:2100196)CMMDS: LeaderBuildHeartbeatMessage:2445: #####################dc: [24230424]:Member[4]:###############0(agent)

    2026-05-27T16:11:40.010Z In(182) vmkernel: cpu33:2100196)CMMDS: CMMDSStateMachineReceiveLoop:1654: #######: Error receiving from #######: Failure
    2026-05-27T16:11:40.010Z In(182) vmkernel: cpu33:2100196)CMMDS: CMMDSStateDestroyNode:708: #######: Destroying node #######: Failed to receive from node
    2026-05-27T16:11:40.010Z In(182) vmkernel: cpu33:2100196)CMMDS: LeaderLostBackup:545: ###### Leader Failover: MUUID ###### old #######
    2026-05-27T16:11:40.010Z In(182) vmkernel: cpu33:2100196)CMMDS: LeaderRemoveNodeFromMembership:8592: ######: Removing node ####### (vsanNodeType: data) from the cluster membership
    2026-05-27T16:11:40.010Z In(182) vmkernel: cpu33:2100196)CMMDS: CMMDSClusterDestroyNodeImpl:262: Destroying node ###### from the cluster db. Last HB received from node - 1729659661149716

     

     

  • Verification of (less /varrun/log/vmkernel.log) of backup host confirms that the host drops its "Backup" status and enters "Discovery" state. In Discovery state, a host resets its view of the cluster and starts scanning the network to figure out who is currently online and who the leader is

    2026-05-27T16:11:39.796Z In(182) vmkernel: cpu59:2100258)DOM: DOMOwner_SetLivenessState:11608: Object 2###### lost liveness [0x45da36c11c40]
    2026-05-27T16:11:40.025Z In(182) vmkernel: cpu18:2100197)CMMDS: BackupRxBackupCease:299: ######c: Received a Cease message from leader
    2026-05-27T16:11:40.025Z In(182) vmkernel: cpu18:2100197)CMMDS: CMMDSLogStateTransition:1838: #####: Transitioning(6######80) from Backup to Discovery: (Reason: Received a Cease message from the leader)
    2026-05-27T16:11:40.025Z In(182) vmkernel: cpu18:2100197)CMMDSNet: CMMDSNet_SetLeader:1330: ######: Updating leader node: old=######0 new=none


    2026-05-27T16:11:19.001Z In(182) vmkernel: cpu14:2100197)CMMDS: CMMDSHeartbeatRequestHBWork:995: 0####: Request heartbeat: Success.
    2026-05-27T16:11:37.001Z In(182) vmkernel: cpu10:2100197)CMMDS: CMMDSHeartbeatCheckHBLogWork:852: #####: Check node returned Failure for node 605ce5c3-b6d8-88ee-3920-bc97e1dd2a50 count 0 fastCount 40 unhealthy 
    2026-05-27T16:11:41.275Z In(182) vmkernel: cpu39:2100197)CMMDS: RejoinRxLeaderHeartbeat:2433: ####### Saw self listed in leader heartbeat
    2026-05-27T16:13:02.251Z In(182) vmkernel: cpu5:2100197)CMMDS: CMMDSHeartbeatRequestHBWork:995:######c30dc: Request heartbeat: Success.
    2026-05-27T16:13:16.775Z In(182) vmkernel: cpu43:2100197)CMMDS: RejoinRxLeaderHeartbeat:2437: ######0dc: No longer listed in heartbeat membership
    2026-05-27T16:13:18.525Z In(182) vmkernel: cpu43:2100197)CMMDS: RejoinRxLeaderHeartbeat:2433: 0e3######: Saw self listed in leader heartbeat
    2026-05-27T16:13:42.501Z In(182) vmkernel: cpu0:2100197)CMMDS: CMMDSHeartbeatRequestHBWork:995: 0#########: Request heartbeat: Success.

     

 

Resolution

  1. Contact the network team to further troubleshoot the ping drops. 

Workaround

    2.If the physical network cannot be corrected immediately, force vSAN traffic onto a healthy redundant path:

    • Administratively disable the physical interface associated with the MTU drops: esxcli network nic down -n vmnicX
    • Verify the cluster stabilizes and nodes rejoin: esxcli network nic list
    • Once the cluster is stable, objects should transition from "Inaccessible" back to "Healthy" as metadata synchronization resumes over the functional path.
Powered by Wolken Software