SDDC Manager add host to cluster fails with error "Vmotion network not found in the network pool"

Products

VMware Cloud Foundation VMware SDDC Manager

Issue/Introduction

Adding a new host to the management WL domain cluster via a json file fails with below error
```
Failed to validate cluster spec for {0} with errors {1}
```

Error in SDDC Manager UI

Generate ESXi Host(s) Addition Input Data

Description    Generate ESXi Host(s) Addition Input Data
Progress Messages    Failed to validate cluster spec for {0} with errors {1}
Error Message: Failed to validate cluster spec for {0} with errors {1}
Remediation Message:
Reference Token: #####

Cause: Vmotion network not found in the network pool.

Error in /var/log/vmware/vcf/domainmanager/domainmanager.log

ERROR [vcf_dm,##############,7aa6] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-6]  [ITI6G4] PUBLIC_CLUSTERSPEC_VALIDATION_FAILED Failed to validate Network parameters for hosts
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to validate Network parameters for hosts
    at com.vmware.evo.sddc.common.services.VmknicConfigGenerator.generateHostVmknicMapping(VmknicConfigGenerator.java:178)
    at com.vmware.vcf.hostmanager.action.header.GenerateAddHostInternalModelAction.execute(GenerateAddHostInternalModelAction.java:169)
    at com.vmware.vcf.hostmanager.action.header.GenerateAddHostInternalModelAction.execute(GenerateAddHostInternalModelAction.java:53)
    at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
    at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)
    at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144)
    at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400)
    at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:520)
    at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)
    at jdk.internal.reflect.GeneratedMethodAccessor758.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:88)
    at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:73)
    at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:68)
    at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:68)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Vmotion network not found in the network pool
    at com.vmware.evo.sddc.common.services.VmknicConfigGenerator.generateHostVmknicMapping(VmknicConfigGenerator.java:90)
    ... 18 common frames omitted

vMotion Network pool in SDDC Manager (Administration > Network settings) has no Default Gateway configured

Cause

vMotion Network in SDDC Network pool has no default gateway address configured

Resolution

Gateway address in vMotion Network in SDDC Manager network pool is a mandatory requirement. If the environment has no default gateway configured or cannot be configured then any valid IP address can be defined as Gateway in vMotion Network in SDDC Manager network pool.

There is no hard requirement for the Gateway address configured in vMotion Network in SDDC Manager network pool to be configured as an actual gateway address in physical switch
There is no hard requirement for the Gateway address configured in vMotion Network in SDDC Manager network pool to be routable if only L2 operations are intended to be performed.

To workaround the issue, update a Gateway address for vMotion network pool in SDDC Manager.

Follow the below steps:

Take snapshot of SDDC Manager VM.
SSH to SDDC Manager with vcf user and su to root

Connect to SDDC Manager platform database

psql -h localhost -U postgres -d platform

Show table listings in narrow format in database tables (optional)
```
\x
```

List the vmotion network from vcf_network table.

select id,gateway,ip_inclusion_ranges,subnet from vcf_network where type='VMOTION';

Sample output

platform=# select id,gateway,ip_inclusion_ranges,subnet from vcf_network where type='VMOTION';
-[ RECORD 1 ]-------+----------------------------------------------
id                  | e1e###############cb
gateway             |
ip_inclusion_ranges | [{"start":"192.0.2.101","end":"192.0.2.250"}]
subnet              | 192.0.2.0

Identify the VMOTION network id on which the host is trying to add and the default gateway address is missing

Update the Default Gateway address for the VMOTION network in vcf_network table in SDDC Manager database

update vcf_network set gateway='<Gateway IP Address>' where id='<vmotion network id noted from step 5>';

Sample

update vcf_network set gateway='192.0.2.254' where id='e1e###############cb';

Check SDDC Manager and confirm Gateway address in updated in vMotion Network in the network pool
SDDC Manager UI > Administration > Network Settings > [Network pool name] > vMotion Network Information
Retry the Add host workflow in SDDC Manager UI

If Gateway cannot be configured in the environment because of limitations in the physical networking then follow the above steps to add the host to Cluster and after task is successful, remove the gateway address from the vMotion network pool

Remove vmotion gateway from vmotion network pool

psql -h localhost -U postgres -d platform -c "update vcf_network set gateway='' where id='<vmotion network id noted from step 5>'"

Sample

psql -h localhost -U postgres -d platform -c "update vcf_network set gateway='' where id='e1e###############cb'"

Remove the vMotion Gateway address from the ESXi host
- vCenter UI > ESXi Host > Enter Maintenance Mode > Configure > Networking > TCP/IP Configuration > vMotion > Edit by clicking the 3 dots > Remove the gateway address from vmkernel gateway > Exit Maintenance Mode