SDDC Manager cluster expansion fails with "Detected out-of-band changes for cluster"

Products

VMware SDDC Manager / VCF Installer VCF Operations

Issue/Introduction

When attempting to add hosts to an existing cluster, the validation fails at the "Validate Platform Infrastructure State for Cluster Expansion workflow" stage with below error:
```
Cluster expansion failed due to: (1) Detected out-of-band changes for cluster: <Cluster_Name>.Host removals:
[<host_A_FQDN>,
<host_B_FQDN>,
].
```

Below error is observed in /var/log/vmware/vcf/domainmanager/domainmanager.log:

ERROR [vcf_dm] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-2472]  CLUSTER_EXPANSION_VALIDATION_FAILED Cluster expansion failed due to: (1) Detected out-of-band changes for cluster: <Cluster_Name> Host removals: [<host_A_FQDN>,<host_B_FQDN>]
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Cluster expansion failed due to: (1) Detected out-of-band changes for cluster: <Cluster_Name>. Host removals: [<host_A_FQDN>,<host_B_FQDN>].
        at com.vmware.vcf.hostmanager.action.ValidatePlatformInfraStateForClusterExpansionAction.execute(ValidatePlatformInfraStateForClusterExpansionAction.java:149)
        at com.vmware.vcf.hostmanager.action.ValidatePlatformInfraStateForClusterExpansionAction.execute(ValidatePlatformInfraStateForClusterExpansionAction.java:36)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:66)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:161)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:147)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:401)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:515)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:128)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
        at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:63)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
 [vcf_dm,] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-2472]  Collected the following errors for task with name ValidatePlatformInfraStateForClusterExpansionAction and [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.evo.sddc.common.services.sddcmanagermessages)]]
[vcf] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-2474]  Invoking task ValidatePlatformInfraStateForClusterExpansionAction.UNDO, Plugin: HostServicePlugin, ParamBuilder null, Input map: {vcEndpoint=vcEndpoint, esxiIds=esxiIds, clusterId=clusterId, skipValidations=ValidateInfraStateForClusterExpansionWorkflow____1__skipValidations, domainId=domainId, clusterSourceId=clusterSourceId
DEBUG [vcf_dm] [c.v.e.s.o.core.recipes.spel.SpelBook,dm-exec]  Field ValidateInfraStateForClusterExpansionWorkflow____1__skipValidations in SpEL expression ValidateInfraStateForClusterExpansionWorkflow____1__skipValidations is not defined. Setting value to null.

Environment

VMware SDDC Manager 9.1
VCF Operations 9.1

Cause

This issue occurs due to a Managed Object Reference (MoRef) mismatch between the SDDC Manager platform database and the vCenter Server inventory. SDDC Manager tracks ESXi hosts using their unique vCenter HostSystem MoRef . If a host is removed from vCenter and subsequently re-added—which frequently happens during host re-imaging, manual inventory cleanup, or certain patching cycles—vCenter assigns a new MoRef . Because the SDDC Manager database retains the stale reference, the validation engine incorrectly concludes that the host has been removed "out-of-band."

Resolution

To resolve this issue, verify the MoRef mismatch and then trigger an inventory synchronization to update the SDDC Manager database with the current vCenter identifiers.

Use SSH to log in to the SDDC Manager appliance as the vcf user and switch to root.

Query the SDDC Manager platform database to identify the stored source_id (MoRef) for the affected hosts.

psql -h localhost -U postgres -d platform -c "SELECT hostname, source_id FROM host WHERE hostname IN ('<host1_fqdn>', '<host2_fqdn>');"

Log in to the vCenter Managed Object Browser (MOB) at https://<vcenter_fqdn>/mob/?moid=group-d1 and navigate to the cluster and host objects to find the current moid. Alternatively, check the host-system managed object directly.
Compare the source_id from the database with the moid from vCenter. If they differ, proceed to the synchronization steps.

To run the brownfield sync we can first Retrieve the Domain ID using:

GET /v1/domains

Example:

{
  "elements": [
    {
      "id": "########-####-####-####-############",
      "name": "<DomainName>",
      "status": "ACTIVE"
    },
  ]
}

Execute a POST request to sync the domain's inventory:

POST /v1/domains/{<domainId_from_step_5>}/synchronizations

Once the synchronization task completes, the host.source_id in the database will be updated. You can then retry the Add Host task in the SDDC Manager UI.