Add Host workflow fails with error SDDC_MANAGER_FAILED_TO_ASSIGN

Products

VMware vSphere ESXi

Issue/Introduction

When executing the "Add Host" workflow in VMware Cloud Foundation (VCF), the process consistently fails at the "Allocate ESXi Host IP Addresses" stage.

The task generates the following explicit error:

This failure triggers an immediate automated rollback (UNDO) action. Review of the domainmanager logs indicates that while trying to allocate a vMotion IP address from the Network Pool, the internal Common Services API abruptly rejects the request with an HTTP 400 (Bad Request) error, even when the UI and database indicate sufficient free IP capacity.

Domainmanager.log:

YYYY-MM-DDT05:21:03.952+0000 DEBUG [vcf_dm,XXXXXXXXXXXXXXXXXXXXX] [c.v.e.s.c.c.v.vsphere.VcManagerBase,XXXXXX] Connecting to https://ESXi-host:443/sdk
.
.
YYYY-MM-DDT05:21:03.988+0000 DEBUG [vcf_dm,XXXXXXXXXXXXXXXXXXXXX] [c.v.e.s.c.v.util.ResponseUtil,dm-exec-10] Build validation response: {"errorCode":"HOST_MANAGEMENT_VNIC_USING_DHCP_VALIDATION.info","arguments":["ESXI-host"],"context":{"severity":"INFO","bundleName":"com.vmware.evo.sddc.common.validation.errors.messages","validation.taskId":"XXXXXXXXXXXXXXXXXXXXXXXXXX"},"message":"ESX Host ESXi-host has static networking configured for the management network","remediationMessage":"ESX Host ESXi-host has DHCP enabled. Configure the ESX Host with static networking for the management network"}
YYYY-MM-DDT05:21:03.991+0000 DEBUG [vcf_dm,XXXXXXXXXXXXXXXXXXXXX] [c.v.e.s.o.c.ProcessingTaskSubscriber] Invoking task ValidateHostsManagementNetworkAction.POSTVALIDATE Description: Validate if ESXi hosts do not use DHCP for the management network, Plugin: HostServicePlugin, ParamBuilder null, Input map: {isSkipFailedHostsEnabled=false, esxiEndpoints=addHosts____6__esxiEndpoints}, Id: XXXXXXXXXXXXXXXXXXXXX.
.
.
YYYY-MM-DDT05:21:04.100+0000 ERROR [vcf_dm,6a338040b1d7c94f4831c35342e52dfc,e47b] [c.v.e.s.c.s.a.n.NetworkPoolAdapterImpl,dm-exec-2] Error occurred fetching all the networks from the network pool XXXXXXXXXXXXXXXXXXXXXXXXXXXXX com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException:
at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.handleResponse(ApiClient.java:788)
at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.execute(ApiClient.java:708)
.
.
YYYY-MM-DDT05:21:04.109+0000 ERROR [vcf_dm,6a338040b1d7c94f4831c35342e52dfc,e47b] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-2] [TASK-ID] SDDC_MANAGER_FAILED_TO_ASSIGN_IP Failed to allocate IPs from networks with error: .
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to allocate IPs from networks with error: .
at com.vmware.evo.sddc.common.services.actions.EsxisHostIpAllocationAction.execute(EsxisHostIpAllocationAction.java:125)
.
.
YYYY-MM-DDT05:21:04.122+0000 DEBUG [vcf_dmXXXXXXXXXXXXXXXXXXXXX] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-2] Collected the following errors for task with name EsxisHostIpAllocationAction and ID XXXXXXXXXXXXXXXXXXXXXXXXX: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.evo.sddc.common.services.sddcmanagermessages)]]
YYYY-MM-DDT05:21:04.126+0000 DEBUG [vcf_dm,6a3380407749e62b80e391ccae4ccec3,71a0] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-2] Invoking task EsxisHostIpAllocationAction.UNDO Description: Allocate ESXi Host IP Addresses, Plugin: HostServicePlugin, ParamBuilder null, Input map: {networkPoolParam=addHosts____6__fibreChannelNetworkPoolParam}, Id: XXXXXXXXXXXXXXXXXXXXX ...
YYYY-MM-DDT05:21:04.127+0000 DEBUG [vcf_dm,6a3380407749e62b80e391ccae4ccec3,71a0] [c.v.e.s.o.c.c.ContractParamBuilder,dm-exec-2] Contract task Allocate ESXi Host IP Addresses input: {"networkPoolParam":{"esxiIds":["XXXXXXXXXXXXXXXXXXXXX"],"networkTypes":["VMOTION"]},"isVvol":false,"_executionUuid":"XXXXXXXXXXXXXXXXXXXXX"}
YYYY-MM-DDT05:21:04.127+0000 DEBUG [vcf_dm,XXXXXXXXXXXXXXXXXXXXX] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-2] Collected the following errors for task with name EsxisHostIpAllocationAction and ID XXXXXXXXXXXXXXXXXXXXX: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.evo.sddc.common.services.sddcmanagermessages)]]
.
.
YYYY-MM-DDT05:21:04.618+0000 INFO [vcf_dm,XXXXXXXXXXXXXXXXXXXXX] [c.v.e.s.t.s.e.util.TaskUtilImpl,XXXXXXX] Updating the state of task with ID XXXXXXXXXXXXXXXXXXXXX to COMPLETED_WITH_FAILURE.

Environment

VMware Cloud Foundation (VCF) 9.0.2

Cause

A data entry issue within the vcf_network table of the SDDC Manager PostgreSQL database. The JSON array storing the available IPs in the free_ip_addresses column contains a syntax error (such as a missing double quotation mark before an IP address), making the JSON array invalid and causing the internal API parser to crash with an HTTP 400 error.

Resolution

To resolve this issue, manually fix the corrupted JSON string syntax directly in the SDDC Manager database:

Step 1:Log into the SDDC Manager appliance as the root user via SSH.

Step 2: Access the SDDC Manager PostgreSQL database instance.

Commnad: psql -h localhost -U postgres -d platform

Step 3: Identify the corrupted entry in the vcf_network table by running a query against the network pool ID associated with the failure.

Command: select * from vcf_network;

Corrupted Entry of the ESXi host would show as: ["10.x.x.x", "10.x.x.x.", "10.x.x.x", "10.x.x.x", "10.x.x.x",10.x.x.x"]

Step 4:Execute an UPDATE query to correct the syntax error inside the free_ip_addresses array (e.g., ensuring all IP strings are enclosed within double quotes correctly):

SQL Command:
UPDATE vcf_network SET free_ip_addresses='["10.x.x.x", "10.x.x.x", "10.x.x.x"]' WHERE id='Network-pool-ID';

Restart the core SDDC Manager services to clear the memory cache and apply the changes:

systemctl restart commonsvcs
systemctl restart domainmanager

Stpe 5: Retry the "Add Host" workflow.