Recover vSAN if the Distributed Switch configuration is altered
search cancel

Recover vSAN if the Distributed Switch configuration is altered

book

Article ID: 433948

calendar_today

Updated On:

Products

VMware vSAN VMware vSphere ESXi

Issue/Introduction

How to recover the vSAN cluster if the vSAN VMKs are on a DVS (Distributed Virtual Switch), but the DVS configuration is altered, causing a vSAN network partition, thereby crashing all the VMs.

It is not possible to revert the change to the DVS if the vCenter VM resides on the vSAN datastore, because it will also go down.

Environment

vSphere 8.0

vSAN 8.0

Cause

Possible causes are:

Removal or change to the DVS uplinks

Removal or change of the port group VLAN

Resolution

A new vSAN VMK must be created on a VSS (Virtual Standard Switch) in order to restore connectivity of the vSAN VMKs.

Do this on each host in the vSAN cluster. See KBs below for additional reference.
 
  1. Remove one vmnic from the DVS for use with the new VSS
    This vmnic must be one that was previously used for vSAN VMK connectivity
    # esxcli network vswitch dvs vmware list
    # esxcfg-vswitch -Q vmnic# -V dvPort_ID_of_vmnic dvSwitch_name

  2. Create a new VSS in the ESX host Client
    Use the vmnic that was removed from the DVS
    Create a new port group with the vSAN network VLAN tag
    Create a new vSAN VMK with temporary IP address

  3. Ping test to test physical network config
    # vmkping -I [new_local_vSAN_VMK] [new_remote_vSAN_VMK]
    example:
    # vmkping -I vmk3 10.1.1.1
    Additional test if jumbo frames are specified:
    # vmkping -I vmk3 -d -s 8972 other_node

  4. Remove the vSAN tag on the old VMK and add the vSAN tag on the new VMK
    # esxcli network ip interface tag add/remove -i vmkX -t VSAN
    # esxcli network ip interface tag get -i vmkX
    example:
    # esxcli network ip interface tag remove -i vmk3 -t VSAN
    # esxcli network ip interface tag add -i vmk4 -t VSAN

  5. Disable member list updates from vCenter
    # esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListUpdates

  6. Configure the unicast agent list, using the new temporary VMK IP addresses
    # esxcli vsan cluster unicastagent list
    # esxcli vsan cluster unicastagent clear
    # esxcli vsan cluster unicastagent add -t node -u <Host_UUID> -U true -a <Host_VSAN_IP>

  7. Check status
    # esxcli vsan cluster get

Additional Information

vCenter network connectivity lost - Recover vCenter network when connected to a Distributed Switch
https://knowledge.broadcom.com/external/article/318719/vcenter-network-connectivity-lost-recov.html

Configuring vSAN Unicast networking from the command line (2150303)
https://knowledge.broadcom.com/external/article/326427/configuring-vsan-unicast-networking-from.html

Configuring Standard vSwitch (vSS) or virtual Distributed Switch (vDS) from the command line in ESXi
https://knowledge.broadcom.com/external/article/326175