VSAN Network Partition After change of IP / DNS Settings.

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• IP and/or DNS Settings were changed on one or more Hosts in the Cluster.
• The affected Host(s) were removed & re-added to vCenter inventory by using the Web Client.
• After this was completed, it was noticed that Host has not joined the Cluster.

When attempting to rejoin the affected Host(s) to the vSAN cluster, it fails with the following error seen in /var/log/vmkernel.log :

CMMDS: CMMDSLogStateTransition:1217: Transitioning(5747011f-e8ee-f43d-9f26-1cxxxx0) from Rejoin to Master: (Reason: Forcing directory liveness)
CMMDS: MasterAddNodeToMembership:5049: Added node 5747011f-e8ee-f43d-9f26-1cxxxxxx50 to the cluster membership
WARNING: CMMDS: CMMDSPublishComplete:123: Failed to publish HOSTNAME 5747011f-e8ee-f43d-9f26-1xxxxx50: Already exists
WARNING: Local node faultDomain ID is changed from 5747011f-e8ee-f43d-9f26-1xxxxxxx0 to 5747011f-e8ee-f43d-9f26-1xxxxxxx0
VSAN Device Monitor: Checking VSAN device latencies and congestion.
CMMDS: MasterCheckNode:5907: Lost contact with 58f61da1-b1dc-8470-8939-14xxxxxxx8
CMMDS: CMMDSStateDestroyNode:608: Destroying node 58f61da1-b1dc-8470-8939-14xxxxxx8: Heartbeat timeout

Environment

VMware vSAN 7.0

VMware vSAN 8.0

Cause

The possible cause for this is duplicate DNS records.

Resolution

To resolve the issue, check DNS settings and find duplicate entries (Two different IP addresses for the same hostname).

With removal of the stale record from DNS, the host will be automatically added to vSAN Cluster.