NSX VIB installation or upgrade fails due to TCP port timeout

Products

• VMware vSphere ESXi
• VMware NSX

Issue/Introduction

When an ESXi host is prepared for NSX or an upgrade to existing NSX VIBs is attempted, the following symptoms may be observed:

• The NSX Manager UI reports the host(s) in an "Installation Failed" state.
• If an upgrade is initialed from SDDC Manager or VCF Operations, the upgrade fails on the Waiting for connection to Managers step.
• When the host is moved to datacenter level, it shows as "Orphaned" on other nodes tab in the NSX Manager UI.
• Connectivity tests from the ESXi host to the NSX Manager IP (####) on TCP ports 443, 1234, or 1235 fail with a timeout. 

nc -zv <NSX Manager IP> 443

nc: connect to <NSX Manager IP> port 443 (tcp) failed: Connection timed out

• nsxcli commands run from the host may fail with an error similar to "Unable to connect to the API service" or return no results, for example:
  
  push host certificate <NSX Manager IP> username admin thumbprint <thumbprint> password <password>

% Push certificate failed: 'Unable to connect to the API service'

get controllers

<date>
 Controller IP    Port     SSL        Status       Is Physical Master   Session State  Controller FQDN          Failure Reason
  
  
  
• ICMP (ping) connectivity between the ESX host and the NSX manager nodes may be successful.

Environment

• VMware NSX 4.x/9.x
• VMware VCF 9.x
• VMware vSphere ESX 8.x/9.x

Cause

This issue can occur for one of the following scenarios:

• Expected Firewall Behavior: On an unprepared ESX host, the required outbound firewall rule for TCP port 443 is not instantiated until the initial NSX VIBs are successfully installed. Therefore, connectivity tests using nc or curl may fail even if the physical network path is open.
• Network Blocking: A physical firewall or security group in your infrastructure is dropping traffic between the ESXi management interface (vmk0) and the NSX Manager cluster IPs.

Resolution

For NSX 4.x

1. Force Remove the Host: In the NSX Manager UI, navigate to System > Fabric > Nodes > Host Transport Nodes. Select the affected host and perform a Delete with the "Force Delete" option enabled to clear the orphaned state.
2. Verify Physical Network: Ensure any firewall between the ESX hosts and the NSX Manager nodes permits traffic on TCP ports 443, 1234, and 1235 between the ESXi host management IPs and all NSX Manager node IPs.
3. Re-add to Cluster: Move the host back into the prepared cluster or trigger a fresh deployment from the NSX UI. The installation process automatically creates the necessary ESXi firewall exceptions once the VIB push begins.

For VCF 9.x

1. Verify Physical Network: Ensure any firewall between the ESX hosts and the NSX Manager nodes permits traffic on TCP ports 443, 1234, and 1235 between the ESXi host management IPs and all NSX Manager node IPs.
2. Restart the failed upgrade task in VCF Operations or SDDC manger.

Additional Information

Unable to connect to TCP ports 443,1234 and 1235 from ESXi host to NSX manager can be used to help diagnose/troubleshoot this issue.

The issue noted in  NSX Host Software Installation fails with alert  “Failed to install software on host. Time out waiting for host to join NSX Manager" during VCF import of a brownfield environment on some hosts may be occurring in a VCF 9.x environment if there is no issue with port connectivity between the ESX hosts and the NSX Manager nodes.

Reference VMware Ports and Protocols for full details on required port connectivity for NSX.