Creating a Workload Domain from SDDC Manager fails at creating NSX transport Node collection stage
search cancel

Creating a Workload Domain from SDDC Manager fails at creating NSX transport Node collection stage

book

Article ID: 390502

calendar_today

Updated On:

Products

VMware SDDC Manager VMware Cloud Foundation

Issue/Introduction

  • Creating a Workload Domain from SDDC Manager fails at creating NSX transport Node collection stage.
  • Error in SDDC Manager UI. 
    Unable to create transport node collection with profile ########-####-####-####-########089d on compute collection #######-####-####-####-########5b81:domain-c9 through nsxmanager nsxmanager.example.com
Remediation Message:
Reference Token: #####
Cause: Failed to realize transport node. Please refer logs.
  • Error in sddc manager /var/log/vmware/vcf/domainmanager/domainamanager.log 
    ERROR [vcf_dm,####################3161,f72d] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-14]  [8D3559] UNABLE_TO_CREATE_TRANSPORT_NODE_COLLECTION Unable to create transport node collection with profile ########-####-####-####-########089d on compute collection #######-####-####-####-########5b81:domain-c9 through nsxmanager nsxmanager.example.com
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Unable to create transport node collection with profile ########-####-####-####-########089d on compute collection #######-####-####-####-########5b81:domain-c9 through nsxmanager nsxmanager.example.com
        at com.vmware.vcf.common.fsm.plugins.nsxt.policy.action.CreateTransportNodeCollectionAction.execute(CreateTransportNodeCollectionAction.java:190)
        at com.vmware.vcf.common.fsm.plugins.nsxt.policy.action.CreateTransportNodeCollectionAction.execute(CreateTransportNodeCollectionAction.java:37)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:520)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)
        at jdk.internal.reflect.GeneratedMethodAccessor966.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
        at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.RuntimeException: Failed to realize transport node. Please refer logs.
  • Further in the /var/log/vmware/vcf/domainmanager/domainamanager.log see that Deployment status as INSTALL_FAILED 
    DEBUG [vcf_dm,####################3161,f72d] [c.v.v.c.f.p.n.p.a.TransportNodeCollectionResolver,dm-exec-14]  Exiting as any of TN status has not changed to IN-PROGRESS or SUCCESS [HostNodeStatus (com.vmware.nsx.model.host_node_status) => {
    configStatus = pending,
    deploymentStatus = INSTALL_FAILED,
    nodeId = ########-####-####-####-#######3b85
}, HostNodeStatus (com.vmware.nsx.model.host_node_status) => {
    configStatus = pending,
    deploymentStatus = INSTALL_FAILED,
    nodeId = ########-####-####-####-########703a
}, HostNodeStatus (com.vmware.nsx.model.host_node_status) => {
    configStatus = pending,
    deploymentStatus = INSTALL_FAILED,
    nodeId = ########-####-####-####-########d10a
}, HostNodeStatus (com.vmware.nsx.model.host_node_status) => {
    configStatus = pending,
    deploymentStatus = INSTALL_FAILED,
    nodeId = ########-####-####-####-########0b19
}]
  • The /var/log/vmware/vcf/domainmanager/domainamanager.log further indicate that vib install from NSX has failed on the hosts because of a connection reset. 
    DEBUG [vcf_dm,####################3161,f72d] [c.v.v.c.n.s.c.c.NsxtManagerTransportNodeOperations,dm-exec-14]  Resolving Fabric Node error [Failed to install software on host. Host 192.168.10.10 not reachable. java.rmi.RemoteException: VI SDK invoke exception:javax.net.ssl.SSLException: Connection reset : java.rmi.RemoteException: VI SDK invoke exception:javax.net.ssl.SSLException: Connection reset] for Host: esxi01.example.com

DEBUG [vcf_dm,####################3161,f72d] [c.v.v.c.n.s.c.c.NsxtManagerTransportNodeOperations,dm-exec-14]  Resolving Fabric Node error [Failed to install software on host. Host 192.168.10.11 not reachable. java.rmi.RemoteException: VI SDK invoke exception:javax.net.ssl.SSLException: Connection reset : java.rmi.RemoteException: VI SDK invoke exception:javax.net.ssl.SSLException: Connection reset] for Host: esxi02.example.com

DEBUG [vcf_dm,####################3161,f72d] [c.v.v.c.n.s.c.c.NsxtManagerTransportNodeOperations,dm-exec-14]  Resolving Fabric Node error [Failed to install software on host. Host 192.168.10.12 not reachable. java.rmi.RemoteException: VI SDK invoke exception:javax.net.ssl.SSLException: Connection reset : java.rmi.RemoteException: VI SDK invoke exception:javax.net.ssl.SSLException: Connection reset] for Host: esxi03.example.com
  • Packet capture between NSX Managers and ESXi host see a reset with a policy deny reason.

Cause

  • All this indicates that there are firewall / port issue which is impacting the connection between NSX Manager and the ESXi host..
  • Required ports for communication between the NSX managers and the hosts are not allowed on the firewall

Resolution

Open the required ports for communication between the NSX managers and the hosts - Refer https://ports.broadcom.com/home/NSX

Powered by Wolken Software