ESXi Transport Node preparation fails to start
search cancel

ESXi Transport Node preparation fails to start

book

Article ID: 371823

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • ESXi Transport node preparation fails with an error
  • Error: 9564: Unable to fetch host hardware and network compute information associated with 1xxxxxc-xxxx-xxxx-xxxx-cxxxxxxxxxx8 from inventory. Either the CM-Inventory Service is not running, or inventory information is missing. Please make the appropriate changes and reapply the configuration
  • Transport node preparation does not even start and fails immediately
  • When you run the esxcli system account list command, you do not see nsx-user created on the ESXi host
  • When you run the esxcli software vib list|grep nsx command on the ESXi host, you do not see any nsx vibs installed
  • Compute manager(vCenter) is registered and status is up from the NSX manager
  • /etc/init.d/cm-inventory status confirms the cm-inventory service is running on all the NSX manager nodes
  • /var/log/cm-inventory/cm-inventory.log on the NSX manager reports error similar to the below

    2024-07-10T05:57:34.150Z  INFO http-nio-127.0.0.1-7443-exec-2 NsxBaseRestController 4446 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Error in API /cm-inventory/api/v1/cm-plugin-container/cm-plugins/4XXXXXX5-XXXX-XXXX-XXXX-6XXXXXXXXXXd?action=get_host_hardware_network_info&target_type=HostSystem&target_id=h
    ost-724503 caused by exception com.vmware.nsx.management.cminv.exceptions.CmInventoryRestRpcException:  {"moduleName":"cm-inventory","errorCode":40500,"errorMessage":"Error in processing CM request: Unable to create connection to cm with id: 4XXXXXX5-XXXX-XXXX-XXXX-6XXXXXXXXXXd"}
    2024-07-10T05:57:34.206Z  INFO http-nio-127.0.0.1-7443-exec-4 CmInventoryFacadeImpl 4446 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Retrieving host properties for cm 4XXXXXX5-XXXX-XXXX-XXXX-6XXXXXXXXXXd and host host-724503
    2024-07-10T05:57:34.208Z  INFO http-nio-127.0.0.1-7443-exec-4 NsxTrustManagerBinding 4446 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Try create  TrustManager of type PKIX
    2024-07-10T05:57:34.225Z  INFO http-nio-127.0.0.1-7443-exec-4 NsxTrustManager 4446 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Trust thumbprint of CN=vcsa01.lab.local,OU=vm,O=vmw,L=bangalore,ST=KRT,C=IN
    2024-07-10T05:57:34.256Z  INFO http-nio-127.0.0.1-7443-exec-4 VcUtilsImpl 4446 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] newest server version is locally-unknown for VC 10.XX.XX.XX, checking max priorVersion
    2024-07-10T05:57:34.256Z  INFO http-nio-127.0.0.1-7443-exec-4 VcUtilsImpl 4446 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Found max prior version v7_0_1_1 of 7.0.1.1 VC 10.XX.XX.XX
    2024-07-10T05:57:34.269Z  INFO http-nio-127.0.0.1-7443-exec-4 NsxTrustManagerBinding 4446 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Try create  TrustManager of type PKIX
    2024-07-10T05:57:34.269Z  INFO http-nio-127.0.0.1-7443-exec-4 VcConnection 4446 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Retrieving service content from VC 10.XX.XX.XX
    2024-07-10T05:57:49.765Z ERROR http-nio-127.0.0.1-7443-exec-4 VcCommunicator 4446 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP40500" level="ERROR" subcomp="cm-inventory"] Unable to create connection to cm with id: 4XXXXXX5-XXXX-XXXX-XXXX-6XXXXXXXXXXd com.vmware.vim.vmomi.client.exception.ConnectionException: https://sdkTunnel:8089/sdk/vimService invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to 10.XX.XX.XX:81 [/10.XX.XX.XX] failed: Connection timed out (Connection timed out)" at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:261) ~[vlsi-client-7.0.1.8343824.jar:?]

Environment

NSX Version 3.1.3.0

Cause

Issue occurs due to the usage of custom ports in VCSA (for example, 4443 for https and 81 for http, for the compute manager connection from NSX manager). Even if connection is open for root user, it is not necessary that port is open for other users(cm-inventory tomcat) on the NSX appliance. Connection to VC is being made through cm-inventory tomcat, which runs with ucminv user

Resolution

Default port is 80 and 443 as mentioned in https://ports.esp.vmware.com/home/NSX-T-Data-Center.

To resolve the issue, run the below commands on all the 3 NSX manager nodes to open custom ports for all the appliance users. 

iptables -A INPUT -p tcp -m tcp --dport 81 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
iptables -A OUTPUT -p tcp -m tcp --dport 81 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
iptables -A OUTPUT -p tcp -m tcp --dport 4443 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 4443 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT

After running above commands on all the 3 NSX manager nodes, NSX should be able to connect to VC in next retry in couple of seconds.

Powered by Wolken Software