NSX-T Host Installation Fails with "Host is not reachable" and "INSTALL_SKIPPED" Status After Host Upgrade using vLCM

Products

• VMware NSX

Issue/Introduction

• The vSphere version of an ESXi host prepared as a transport node was upgraded using vLCM.
  
  
• Installation of NSX VIBs on some hosts in the cluster failed, and the NSX Manager UI displayed the error "INSTALL_SKIPPED".
  
  
• The following error message is repeatedly logged in /var/log/syslog on the NSX Manager:
  
  YYYY-MM-DDTHH:MM:SS.NNNZ ##### NSX #### FABRIC [nsx@#### comp="nsx-manager" level="INFO" subcomp="manager"] FN ########-####-####-####-############ installation status = Waiting for fabric-node installation failure to be fixed: Host <hostname> is not reachable. Check host's connectivity and health in vCenter and retry., statelessHost = false, forceSync = false
YYYY-MM-DDTHH:MM:SS.NNNZ ##### NSX #### FABRIC [nsx@#### comp="nsx-manager" level="INFO" subcomp="manager"] Paused configuring TransportNode TransportNode/########-####-####-####-############. Waiting for fabric-node installation failure to be fixed: Host <hostname> is not reachable. Check host's connectivity and health in vCenter and retry.

Environment

VMware NSX

Cause

The NSX Manager repeatedly fails to install NSX components on the host because it determines the host is "not reachable". 
This unreachability, often indicated by the message "Failed to install software on host. Host [hostname] is not reachable. Check host's connectivity and health in vCenter and retry," is frequently a result of a temporary connection error with the Management Plane that occurred during the host upgrade process using vLCM. 
The host status is internally marked as INSTALL_SKIPPED or UNINSTALL_SKIPPED due to these failed attempts in a vLCM integrated setup.

Resolution

To resolve the host's status and successfully install the NSX components:

1. On the NSX Manager, go to SystemFabricHostsClusters.
   
   
2. Identify the failed cluster to view the error state. Click the error link to open a popup window.
   
   
3. If the cluster is in Install Failed state, click Resolve at the to initiate transport node profile realization on the cluster.
   
   Important:
   With the cluster in Install Failed state, first try to resolve the remediation issues at the cluster and then try to remediate individual hosts.
   If you overlook cluster-level errors and directly try to remediate host-level errors, the UI does not allow you to perform any remediation action at the host-level.
   
   
4. If one or more hosts failed but the cluster remediation status is Success, then navigate to the failed host and click Resolve to remediate hosts.
   
   
5. You can also try to realize the transport node profile on the cluster by executing the following API command.
   
   POST /api/v1/transport-node-collections/{tnc id}?action=retry_profile_realization
   
   This API command re-triggers the transport node profile on the cluster.

Additional Information

The remediation steps align with best practices for addressing host preparation failures in vLCM integrated environments.

Please refer to the vSphere Lifecycle Manager Failed to Prepare a Host for NSX Networking for more details on resolving host preparation issues.