Failed to join the host in VSAN cluster while adding a host back to the vSAN cluster

Products

• VMware vSAN

Issue/Introduction

Symptoms:

In the vSphere UI, adding the host back to its vSAN cluster, you see the warning similar to:​

Host is in a Virtual SAN cluster but does not have Virtual SAN service enabled

Output:

# esxcli vsan cluster get
Virtual SAN Clustering is not enabled on this host

# esxcli vsan cluster join -u 5243793c-7c36-ed29-d3f1-ce86f210####
Failed to join the host in VSAN cluster (Disk with VSAN uuid 00000000-0000-0000-0000-000000000000 failed to appear in VSAN directory)

In the /var/log/vmkernel.log file, you see the entries similar to: 

CMMDS: CMMDSLogStateTransition:1217: Transitioning(593a455a-2f60-d118-fb1c-9418828####) from Master to Invalid: (Reason: State machine deinitialization)
CMMDS: MasterRemoveNodeFromMembership:5079: Removing node 593a455a-2f60-d118-fb1c-######### from the cluster membership
PLOG: PLOGProbeDevice:5229: Probed plog device <naa.5000cca03210cfd7:1> 0x430756b81160 exists.. continue with old entry
PLOG: PLOGProbeDevice:5229: Probed plog device <naa.5000cca03210cfd7:2> 0x430756b7fc20 exists.. continue with old entry
WARNING: PLOG: PLOGValidateDisk:2546: Possibly corrupt metadata read from disk naa.5000cca03210cfd7, checksum mismatch - expected 0xe09d656a84b98472, got 0x0 disk remains unpublished
Vol3: 1023: Couldn't read volume header from naa.5000cca03210cfd7:1: I/O error
FSS: 5353: No FS driver claimed device 'naa.5000cca03210cfd7:1': No filesystem on the device

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSAN 6.x

VMware vSAN 7.x

VMware vSAN 8.x

Cause

This is caused when the metadata on the disk is corrupt.

Resolution

To resolve this issue, delete the previous partitions and add the host back to the vSAN cluster (any previous data on this disk will then be overwritten).

To delete the previous partitions and add the host back to the vSAN cluster:

1. Navigate to the vSAN cluster.

2. Click the Configure tab.

3. Under vSAN, click Disk Management.

4. Select a host to view the list of available devices.

5. From the Show drop-down menu, select Ineligible.

6. Select a device from the list.

Option	Description
vSphere Client	Click Erase partitions.
vSphere Web Client	Click the Erase partitions icon.

From the command line:

1. Run command to see the current partition table on the disk:​

# partedUtil getptbl /vmfs/devices/disks/naa.5000cca03210cfd7
gpt
145921 255 63 2344225968
1 2048 6143 381CFCCC728811E092EE000C2911D0B2 vsan 0
2 6144 2344225934 77719A0CA4A011E3A47E000C29745A24 virsto 0

 

2. Delete the existing partitions on the disk:

 

# partedUtil delete /dev/disks/naa.5000cca03210cfd7 1
# partedUtil delete /dev/disks/naa.5000cca03210cfd7 2

3. Add the host back to the vSAN cluster.

If these options does not work, please open a support request with VMware by Broadcom for further assistance.