SDDC Cluster Creation Fails with "vSAN Cluster UUID Mismatch" and "Cannot skip ESXi Host(s)... minimum is 3" Errors

Products

VMware SDDC Manager VMware Cloud Foundation VCF Operations

Issue/Introduction

When attempting to create an SDDC cluster in VMware Cloud Foundation (VCF), the operation may fail at the "Add ESXi Host to vSphere Cluster" task.
The primary errors observed are:
- A vSAN cluster operation failure due to UUID mismatch when adding hosts to the vSphere/vSAN cluster.
- A VCF-level orchestration error indicating that hosts cannot be skipped because the remaining host count would fall below the required minimum of 3 for a vSAN cluster.

Error in VCF Ops or SDDC Manager UI

Message: Hosts esxhost1.example.com, esxhost2.example.com cannot be added to the cluster domain-c###### 
Remediation Message:
Reference Token: K####5
Cause: Cannot skip 2 ESXi Host(s) ([esxhost1.example.com,esxhost2.example.com]) as only 1 ESXi host(s) would remain and the minimum

Error in SDDC manager domainmanager.log: /var/log/vmware/vcf/domainmanager/domainmanager.log

DEBUG [vcf_dm,69######################41,1b64] [c.v.e.s.c.h.i.HostPreparationServiceImpl,dm-exec-20]  ESXi host esxhost3.example.com joined vSAN cluster 6139####-####-####-####-########6e4c. Result: true

ERROR [vcf_dm,69#####################2f,1241] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-24]  [K####5] VSPHERE_HOSTS_CANNOT_BE_ADDED_TO_CLUSTER Hosts esxhost1.example.com, esxhost2.example.com cannot be added to the cluster domain-c######
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Hosts esxhost1.example.com, esxhost2.example.com cannot be added to the cluster domain-c######

Caused by: com.vmware.evo.sddc.common.core.error.InvalidStateException: Cannot skip 2 ESXi Host(s) ([esxhost1.example.com, esxhost2.example.com]) as only 1 ESXi host(s) would remain and the minimum is 3

WARN  [vcf_dm,69#####################2f,1241] [c.v.v.v.c.h.i.HttpProtocolBindingBase,dm-exec-24]  Asynchronous execution requested but no Executor configured. The request will be executed as synchronous one.
ERROR [vcf_dm,69#####################2f,1241] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-24]  Task: (MOR:task-1######4) (Name:moveInto) Entity: (MOR:domain-c######) (Name:TEST-CLUSTER-01) is failed
com.vmware.vim.binding.vim.fault.VsanClusterUuidMismatch: The vSAN host cannot be moved to the destination cluster: vSAN cluster UUID mismatch (host: c443####-####-####-####-########83f5, destination: 6139####-####-####-####-########6e4c

WARN  [vcf_dm,69#####################2f,1241] [c.v.v.v.c.h.i.HttpProtocolBindingBase,dm-exec-24]  Asynchronous execution requested but no Executor configured. The request will be executed as synchronous one.
ERROR [vcf_dm,69#####################2f,1241] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-24]  Task: (MOR:task-1######0) (Name:moveInto) Entity: (MOR:domain-c######) (Name:TEST-CLUSTER-01) is failed
com.vmware.vim.binding.vim.fault.VsanClusterUuidMismatch: The vSAN host cannot be moved to the destination cluster: vSAN cluster UUID mismatch (host: 7a76####-####-####-####-########42af, destination: 6139####-####-####-####-########6e4c).

Environment

VMware Cloud Foundation
VMware SDDC Manager
VMware Cloud Foundation Ops 9.0

Cause

The failure is caused by the target ESXi hosts still being members of a pre-existing or remnant vSAN cluster, leading to a vSAN Cluster UUID Mismatch when the VCF workflow attempts to move them into the new vSAN cluster.

Resolution

The resolution involves manually leaving the remnant vSAN cluster on the affected ESXi hosts and removing all associated vSAN disk group metadata before retrying the VCF cluster creation task.

Steps to follow:

Perform the following steps on each of the failed ESXi hosts

SSH to the affected ESXi host with root.
List the active vSAN cluster status to confirm the hosts are still members of a cluster:
```
esxcli vsan cluster get
```
List all vSAN-claimed storage devices and their UUIDs:
```
esxcli vsan storage list | grep "VSAN UUID"
```
Remove all vSAN-claimed storage devices one by one using their respective UUIDs. Repeat this command for every UUID found in the previous step:
```
esxcli vsan storage remove -u <VSAN_UUID>
```
Caution: In some cases, you may encounter an error like Unable to remove device: Can not remove last HDD. If this occurs, continue removing the other UUIDs first and then retry the remaining one(s).
Leave the existing vSAN cluster once all storage devices are successfully removed:
```
esxcli vsan cluster leave
```
Verify that vSAN clustering is disabled on the host:
```
esxcli vsan cluster get
```
The output should be: vSAN Clustering is not enabled on this host
Retry the failed VCF task in SDDC Manager.

The "Add ESXi Host to vSphere Cluster" task should now succeed as the hosts are no longer reporting a conflicting vSAN Cluster UUID.