Error: SDDC cluster creation fails with vSAN enabled on ESXi hosts

Products

• VMware SDDC Manager / VCF Installer
• VMware Cloud Foundation
• VCF Operations

Issue/Introduction

• SDDC Manager cluster creation workflows fail during host validation when using commissioned or re-imaged hosts that contain remnant vSAN metadata.
• Workflow fails at the "Validate Provided Cluster Specification" task.
• Error Message: "Failed to validate cluster spec."
• Remediation Message: "Cannot skip 3 ESXi Host(s) ... as only 0 ESXi host(s) would remain and the minimum is 3."
• The domainmanager.log contains the following entry: ERROR [vcf_dm,####,####] [c.v.v.h.s.v.HostValidatorUtil] ESXi Host has vSAN enabled.
• When attempting to create an SDDC cluster in VMware Cloud Foundation (VCF), the operation may fail at the "Add ESXi Host to vSphere Cluster" task.
• The primary errors observed are:
  • A vSAN cluster operation failure due to UUID mismatch when adding hosts to the vSphere/vSAN cluster.
  • A VCF-level orchestration error indicating that hosts cannot be skipped because the remaining host count would fall below the required minimum of 3 for a vSAN cluster.
    
    
• Error in VCF Ops or SDDC Manager UI

  Message: Hosts esxhost1.example.com, esxhost2.example.com cannot be added to the cluster domain-c###### 
Remediation Message:
Reference Token: K####5
Cause: Cannot skip 2 ESXi Host(s) ([esxhost1.example.com,esxhost2.example.com]) as only 1 ESXi host(s) would remain and the minimum

• Error in SDDC manager domainmanager.log: /var/log/vmware/vcf/domainmanager/domainmanager.log
  
  DEBUG [vcf_dm,69######################41,1b64] [c.v.e.s.c.h.i.HostPreparationServiceImpl,dm-exec-20]  ESXi host esxhost3.example.com joined vSAN cluster 6139####-####-####-####-########6e4c. Result: true
  

  ERROR [vcf_dm,69#####################2f,1241] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-24]  [K####5] VSPHERE_HOSTS_CANNOT_BE_ADDED_TO_CLUSTER Hosts esxhost1.example.com, esxhost2.example.com cannot be added to the cluster domain-c######
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Hosts esxhost1.example.com, esxhost2.example.com cannot be added to the cluster domain-c######

  Caused by: com.vmware.evo.sddc.common.core.error.InvalidStateException: Cannot skip 2 ESXi Host(s) ([esxhost1.example.com, esxhost2.example.com]) as only 1 ESXi host(s) would remain and the minimum is 3

  WARN  [vcf_dm,69#####################2f,1241] [c.v.v.v.c.h.i.HttpProtocolBindingBase,dm-exec-24]  Asynchronous execution requested but no Executor configured. The request will be executed as synchronous one.
ERROR [vcf_dm,69#####################2f,1241] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-24]  Task: (MOR:task-1######4) (Name:moveInto) Entity: (MOR:domain-c######) (Name:TEST-CLUSTER-01) is failed
com.vmware.vim.binding.vim.fault.VsanClusterUuidMismatch: The vSAN host cannot be moved to the destination cluster: vSAN cluster UUID mismatch (host: c443####-####-####-####-########83f5, destination: 6139####-####-####-####-########6e4c

  WARN  [vcf_dm,69#####################2f,1241] [c.v.v.v.c.h.i.HttpProtocolBindingBase,dm-exec-24]  Asynchronous execution requested but no Executor configured. The request will be executed as synchronous one.
ERROR [vcf_dm,69#####################2f,1241] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-24]  Task: (MOR:task-1######0) (Name:moveInto) Entity: (MOR:domain-c######) (Name:TEST-CLUSTER-01) is failed
com.vmware.vim.binding.vim.fault.VsanClusterUuidMismatch: The vSAN host cannot be moved to the destination cluster: vSAN cluster UUID mismatch (host: 7a76####-####-####-####-########42af, destination: 6139####-####-####-####-########6e4c).

Environment

• VCF 5.x
• VCF 9.x
• VCF Operations 9.x

Cause

The failure is caused by the target ESXi hosts still being members of a pre-existing or remnant vSAN cluster, leading to a vSAN Cluster UUID Mismatch when the VCF workflow attempts to move them into the new vSAN cluster.

Resolution

The resolution involves manually leaving the remnant vSAN cluster on the affected ESXi hosts and removing all associated vSAN disk group metadata before retrying the VCF cluster creation task.

Steps to follow:

Perform the following steps on each of the failed ESXi hosts

1. SSH to the affected ESXi host with root.
2. List the active vSAN cluster status to confirm the hosts are still members of a cluster:

   esxcli vsan cluster get

3. List all vSAN-claimed storage devices and their UUIDs:

   esxcli vsan storage list | grep "VSAN UUID"

4. Remove all vSAN-claimed storage devices one by one using their respective UUIDs. Repeat this command for every UUID found in the previous step:
   
   esxcli vsan storage remove -u <VSAN_UUID>
   
   Caution: In some cases, you may encounter an error like Unable to remove device: Can not remove last HDD. If this occurs, continue removing the other UUIDs first and then retry the remaining one(s).
   
   
5. Leave the existing vSAN cluster once all storage devices are successfully removed:
   
   esxcli vsan cluster leave
   
   
6. Verify that vSAN clustering is disabled on the host:

   esxcli vsan cluster get

   The output should be: vSAN Clustering is not enabled on this host
   
   
7. Retry the failed VCF task in SDDC Manager. 

The "Add ESXi Host to vSphere Cluster" task should now succeed as the hosts are no longer reporting a conflicting vSAN Cluster UUID.