Duplicate vSAN IP address cause host to become Network partitioned

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• vSAN host frequently becomes Network partitioned, then after sometime the host joins back without any intervention. 
• /var/run/log/vsansystem.logs will report the change in membership count:

YYYY-MM-DDTHH:MM:SS.###Z cpu0:2098825)CMMDS: CMMDSUtil_PrintArenaEntry:100: ########-####-####-####-##########: [60579888]:Adding a new Membership entry (########-####-####-####-##########) with 5 members:
YYYY-MM-DDTHH:MM:SS.###Z cpu0:2098825)CMMDS: CMMDSUtil_PrintArenaEntry:100: ########-####-####-####-##########: [60644183]:Adding a new Membership entry (########-####-####-####-##########) with 1 members: 
YYYY-MM-DDTHH:MM:SS.###Z cpu0:2098825)CMMDS: CMMDSUtil_PrintArenaEntry:100: ########-####-####-####-##########: [60697647]:Adding a new Membership entry (########-####-####-####-##########) with 5 members:

• /var/run/log/vmkernel.log reports that another device in the network have the same IP address of the VMkernel adaptor that is tagged for the vSAN traffic. It will also provide the MAC address of the other device.

YYYY-MM-DDTHH:MM:SS.###Z cpu78:2098609)Tcpip_Vmk: 124: arp: aa:bb:cc:dd:ee:ff is using my IP address 192.168.xx.xx on vmkX!
YYYY-MM-DDTHH:MM:SS.###Z cpu78:2098609)Tcpip_Vmk: 124: arp: aa:bb:cc:dd:ee:ff is using my IP address 192.168.xx.xx on vmkX!
YYYY-MM-DDTHH:MM:SS.###Z cpu78:2098609)Tcpip_Vmk: 124: arp: aa:bb:cc:dd:ee:ff is using my IP address 192.168.xx.xx on vmkX!
YYYY-MM-DDTHH:MM:SS.###Z cpu78:2098609)Tcpip_Vmk: 124: arp: aa:bb:cc:dd:ee:ff is using my IP address 192.168.xx.xx on vmkX!
YYYY-MM-DDTHH:MM:SS.###Z cpu78:2098609)Tcpip_Vmk: 124: arp: aa:bb:cc:dd:ee:ff is using my IP address 192.168.xx.xx on vmkX!

• VMkernel adaptor used for vSAN traffic can be identified using the below command.

# esxcli vsan network list

Interface:
   VmkNic Name: vmkX
   IP Protocol: IP
   Interface UUID: ########-####-####-####-##########
   Agent Group Multicast Address: ###.#.#.#
   Agent Group IPv6 Multicast Address: ####::#:#:#
   Agent Group Multicast Port: 23451
   Master Group Multicast Address: ###.#.#.#
   Master Group IPv6 Multicast Address: ####::#:#:#
   Master Group Multicast Port: 12345
   Host Unicast Channel Bound Port: 12321
   Data-in-Transit Encryption Key Exchange Port: 0
   Multicast TTL: 5
   Traffic Type: vsan

Environment

VMware vSAN 6.x

VMware vSAN 7.x

VMware vSAN 8.x

Cause

This issue occurs if any other device in the network have the same IP address of the VMkernel adaptor that is tagged for the vSAN traffic.

 

Resolution

Identify the other device in the network with the same IP address of the vSAN VMkernel adaptor. Change its IP address to avoid the conflict.

Additional Information

Troubleshoot duplicate IP without shutting down known device