vSAN cluster partition after vSAN untagged from vmkernel network adapter

Products

• VMware vSAN

Issue/Introduction

• The vSAN cluster reports a "Network Partitioned" state.
• Running esxcli vsan cluster get shows a Sub-Cluster Member Count of 1, despite multiple hosts being present in the environment.

 

Cluster Information

Enabled: true

Current Local Time: YYYY-MM-DDThh:mm:ss

Local Node UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Local Node Type: NORMAL

Local Node State: MASTER

Local Node Health State: HEALTHY

Sub-Cluster Master UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Sub-Cluster Backup UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Sub-Cluster UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Sub-Cluster Membership Entry Revision: #

Sub-Cluster Member Count: 1

Sub-Cluster Member UUIDs: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Sub-Cluster Member HostNames: <ESXi host_NAME>

Sub-Cluster Membership UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Unicast Mode Enabled: true

Maintenance Mode State: OFF

Config Generation: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx YYYY-MM-DDThh:mm:ss

Mode: REGULAR

VSAN ESA Enabled: false

 

• From the vSphere under ESXi host > Configure > vmkernel adapter > VSAN might not be tagged on vmkernel adapter.

 

• A decrease in the vSAN node count may be recorded in the /var/run/log/vsansystem.log.

 

YYYY-MM-DDThh:mm:ss vsansystem[2103526]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSMembershipUpdate-2b84] Complete, nodeCount: 1, runtime info: (vim.vsan.host.VsanRuntimeInfo) {

YYYY-MM-DDThh:mm:ss  vsansystem[2103389]: -->    membershipList = (vim.vsan.host.MembershipInfo) [

YYYY-MM-DDThh:mm:ss   vsansystem[2103389]: -->       (vim.vsan.host.MembershipInfo) {

YYYY-MM-DDThh:mm:ss  vsansystem[2103389]: -->          nodeUuid = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",

YYYY-MM-DDThh:mm:ss  vsansystem[2103389]: -->          hostname = "<ESXI host_name>"

YYYY-MM-DDThh:mm:ss  vsansystem[2103389]: -->       }

YYYY-MM-DDThh:mm:ss  vsansystem[2103389]: -->    ]

 

 

• The var/run/log/vobd.log may report a 'vnic deleted' event, confirming that vSAN is no longer bound to the required VMkernel adapter. This mismatch in the networking configuration prevents the host from participating in the vSAN cluster."

 

YYYY-MM-DDThh:mm:ss  vobd[2097956]:  [vSANCorrelator] 3042565106538us: [vob.vsan.net.no.connectivity] vSAN is no longer using vmknic vmk#. There are no vSAN vmknics remaining.

YYYY-MM-DDThh:mm:ss  vobd[2097956]:  [vSANCorrelator] 3042839981191us: [esx.audit.vsan.net.vnic.deleted] vSAN vnic deleted

 

• The /var/run/log/vmkernel.log indicates that the vSAN daemon has been shut down, which typically results in the host losing its connection to the cluster.

 

YYYY-MM-DDThh:mm:ss  vmkernel: cpu##:10461253)Deactivating Daemon vsan-shutdown.

YYYY-MM-DDThh:mm:ss  vmkernel: cpu##:10461259)RDT: RDTDestroyNetwork:1385: Destroyed RDT Network: vmknic = vmk#, addr = ###.##.###.##, traffic type mask 0x3, witness networks 0, allow reconfig 1

YYYY-MM-DDThh:mm:ss  vmkernel: cpu#:2099979)CMMDSNet: CMMDSNet_ContextRemoveInterface:780: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: Removing Interface for context xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: vmk#

YYYY-MM-DDThh:mm:ss  vmkernel: cpu#:2099979)CMMDS: CMMDSVSIUpdateNetworkCbk:4035: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx: DELETE of interface vmk# with cmmds (Success).

 

Environment

VMware vSAN 8.x

VMware vSAN 9.x

 

Cause

 

• The vSAN management service (vsanmgmtd) fails to synchronize cluster membership (CMMDS) across all ESXi hosts, often following certificate updates or global service restarts.
• The vSAN traffic tag is removed from the VMkernel adapter (e.g., vmk#). This can happen if the adapter was deleted and recreated without re-applying the vSAN tag or manually removed the vSAN tag from the vmkernel adapter.

Resolution

• If the vSAN tag is missing from the interface intended for vSAN traffic, tag it from ESXi host > Configure > VMkernel adapters >  Select specific adapter and enable vSAN tag on it.
• In case VSAN kernel is on a vDS and vcenter is not reachable, add the missing tag using ESXi host CLI
  #esxcli network ip interface tag add -i vmk# -t VSAN
  
  
•  Restart vSAN Management Service from the ESXi host CLI.

/etc/init.d/vsanmgmtd restart