SDDC Manager New Cluster Creation fails with error "Failed to validate cluster spec" due to stale vSAN metadata on ESXi hosts
search cancel

SDDC Manager New Cluster Creation fails with error "Failed to validate cluster spec" due to stale vSAN metadata on ESXi hosts

book

Article ID: 449756

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer VMware Cloud Foundation

Issue/Introduction

  • Adding new cluster to existing Workload domain fails with error

    Message: Failed to validate cluster spec
    Cause: Cannot invoke "com.vmware.vcf.clustermanager.fsm.params.FetchHostDetailsParam.getEsxilds()" because "hostDetailsParam" is null



  • /var/log/vmware/vcf/domainmanager/domainmanager.log on VCF Ops shows below errors,

YYYY-MM-DDTHH:MM:SS WARN  [vcf_dm,################################,####] [c.v.v.c.f.a.ValidateClusterSpecAction,dm-exec-13]  Validate Cluster Spec failed for ESXi hosts <ESXi_FQDN>. Skipping failed hosts. Reference Token: ######.

YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,################################,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-13]  [######] PUBLIC_CLUSTERSPEC_VALIDATIONS_FAILED Failed to validate cluster spec
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to validate cluster spec
        at com.vmware.vcf.clustermanager.fsm.actions.ValidateClusterSpecAction.skipFailedHosts(ValidateClusterSpecAction.java:274)
                at com.vmware.vcf.clustermanager.fsm.actions.ValidateClusterSpecAction.validateIfHostsAreVsanReady(ValidateClusterSpecAction.java:231)
        at com.vmware.vcf.clustermanager.fsm.actions.ValidateClusterSpecAction.execute(ValidateClusterSpecAction.java:141)
        at com.vmware.vcf.clustermanager.fsm.actions.ValidateClusterSpecAction.execute(ValidateClusterSpecAction.java:50)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:520)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)
        at jdk.internal.reflect.GeneratedMethodAccessor423.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
        at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.vmware.evo.sddc.common.core.error.InvalidStateException: Cannot skip 1 ESXi Host(s) ([<ESXi_FQDN>]) as only 2 ESXi host(s) would remain and the minimum is 3

  • /var/run/log/vsanmgmt.log on ESXi affected host shows below error,

    YYYY-MM-DDTHH:MM:SS In(14) vsand[2130701]: [opID=MainThread statsdaemon::CheckAndIncreaseStatsObjectSize] CheckAndIncreaseStatsObjectSize: objExisting is False, statsDbUuid is None, currentObjSize is 0
    YYYY-MM-DDTHH:MM:SS In(14) vsand[2130701]: [opID=MainThread statsdaemon::HandleRoleChanges] statsdaemon handlers with false callbackInstalled are {'VsanAnalyticsHostEventsMonitor.CmmdsEntryDefaultHandler', 'VsanHealthDaemon.CmmdsDiskDecomStateHandler', 'VsanHealthDaemon.CmmdsEntryDefaultHandler', 'VsanHealthDaemon.CmmdsDiskHealthStatusHandler'}
    YYYY-MM-DDTHH:MM:SS Er(11) vsand[2130701]: [opID=MainThread statsdaemon::run] Handle Changes
    YYYY-MM-DDTHH:MM:SS Er(11)[+] vsand[2130701]: Traceback (most recent call last):
    YYYY-MM-DDTHH:MM:SS Er(11)[+] vsand[2130701]:   File "/usr/lib/vmware/vsan/perfsvc/statsdaemon.py", line 1911, in run
    YYYY-MM-DDTHH:MM:SS Er(11)[+] vsand[2130701]:   File "/usr/lib/vmware/vsan/perfsvc/statsdaemon.py", line 1669, in HandleRoleChanges
    YYYY-MM-DDTHH:MM:SS Er(11)[+] vsand[2130701]: PyCppVmomi.vim.fault.VsanFault: (vim.fault.VsanFault) {
    YYYY-MM-DDTHH:MM:SS Er(11)[+] vsand[2130701]:   msg = 'Timed out when installing CMMDS subscription',
    YYYY-MM-DDTHH:MM:SS Er(11)[+] vsand[2130701]:   faultMessage = (vmodl.LocalizableMessage) [
    YYYY-MM-DDTHH:MM:SS Er(11)[+] vsand[2130701]:     (vmodl.LocalizableMessage) {
    YYYY-MM-DDTHH:MM:SS Er(11)[+] vsand[2130701]:       key = 'com.vmware.vsan.perfsvc.fault.cmmds.subscription.timeout',
    YYYY-MM-DDTHH:MM:SS Er(11)[+] vsand[2130701]:       message = 'Timed out when installing CMMDS subscription'
    YYYY-MM-DDTHH:MM:SS Er(11)[+] vsand[2130701]:     }
    YYYY-MM-DDTHH:MM:SS Er(11)[+] vsand[2130701]:   ]
    YYYY-MM-DDTHH:MM:SS Er(11)[+] vsand[2130701]: }

  • /var/run/log/vmkernel.log on affected ESXi host logs shows below information that vSAN is configured,

    YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu47:####### opID=########)Config: 725: "VsanPersonalityConfigured" = 2, Old Value: 0, (Status: 0x0)

Environment

VCF 5.x

Cause

During a previous failed cluster deployment, the ESXi host was partially configured for vSAN. Although the task failed, stale vSAN personality and CMMDS (Cluster Monitoring, Membership, and Directory Service) metadata remain on the host. When a new cluster creation task is initiated, SDDC Manager detects the host is already "vSAN enabled" or in an inconsistent state, causing the validation to fail.

Resolution

To resolve this, the impacted ESXi host must be completely cleared of all vSAN metadata and returned to a "clean" state before being commissioning into SDDC Manager.

  1. Decommission the impacted host(s): In the SDDC Manager UI, navigate to Inventory > Hosts and decommission the host(s) reporting the validation error.

  2. Re-image the ESXi host: Perform a fresh installation of ESXi on the impacted host to ensure all local partitions, vSAN metadata, and configuration residue are removed.

  3. Commission the host(s): Once re-imaged and configured with the correct networking/passwords, commission the host(s) back into the SDDC Manager inventory.

  4. Retry Cluster Creation: Initiate the Add Cluster workflow again. The validation should now pass as the hosts are in a clean, non-vSAN state

Impact/Risks:

  • Decommissioning and re-imaging will erase all data and configurations on the local ESXi host. Ensure no critical data resides on the host before proceeding.

  • Ensure the host is re-imaged with a VCF-supported ESXi version that matches the current SDDC Manager software BOM.

Powered by Wolken Software