Add cluster failure with error "Unable to create transport node collection with profile"

Products

• VMware Cloud Foundation

Issue/Introduction

/var/log/vmware/vcf/domainmanager/domainmanager.log:

 

[YYYY-MM-DDTHH:MM:SS] ERROR [vcf_dm,67dd2c3699d76d2619b31684f833cf52,3856] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-14]  [R4HBHK] UNABLE_TO_CREATE_TRANSPORT_NODE_COLLECTION Unable to create transport node collection with profile "ID" on compute collection "ID":domain-ID through nsxmanager "NSX-T FQDN"
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Unable to create transport node collection with profile "ID" on compute collection "ID":domain-ID through nsxmanager "NSX-T FQDN"
        at com.vmware.vcf.common.fsm.plugins.nsxt.policy.action.CreateTransportNodeCollectionAction.execute(CreateTransportNodeCollectionAction.java:190)
        at com.vmware.vcf.common.fsm.plugins.nsxt.policy.action.CreateTransportNodeCollectionAction.execute(CreateTransportNodeCollectionAction.java:36)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:520)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)
        at jdk.internal.reflect.GeneratedMethodAccessor1344.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
        at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: Failed to realize transport node. Please refer logs.
        at com.vmware.vcf.common.fsm.plugins.nsxt.policy.services.impl.PolicyTransportNodeCollectionServiceImpl.checkClusterState(PolicyTransportNodeCollectionServiceImpl.java:127)
        at com.vmware.vcf.common.fsm.plugins.nsxt.policy.action.CreateTransportNodeCollectionAction.lambda$execute$1(CreateTransportNodeCollectionAction.java:154)
        at com.vmware.vcf.common.fsm.plugins.nsxt.helpers.NsxtCommonOperations.waitForState(NsxtCommonOperations.java:148)
        at com.vmware.vcf.common.fsm.plugins.nsxt.policy.action.CreateTransportNodeCollectionAction.execute(CreateTransportNodeCollectionAction.java:148)
        ... 16 common frames omitted

 

NSX-T error identified from the logs -
Failed to install software on host. Failed to install software on host.<Hostname>: java.rmi.RemoteException: [LiveInstallationError] VMware_bootbank_nsx-mpa_3.1.3.0.0-7.0.18103256: Error in running [/etc/init.d/nsx-mpa start install]: Return code: 1 Output: Unable to setmemconfig for mpa host/vim/vmvisor/mpa Resource pool ‘host/vim/vmvisor/mpa’ released. Waiting for /usr/lib64/vmware/nsx-mpa/m

Resolution

• Reboot all the hosts in the cluster 
• Configure TN manually from NSXT UI 
• Restart the failed add cluster workflow