NSX Installation on an ESXi Host fails with an error "Failed to install software on host. Failed to download NSX components on host. Check the host connectivity and if "/tmp" folder has enough space"

Products

• VMware NSX

Issue/Introduction

• When attempting to install NSX on an ESXi host, it fails with error "Failed to install software on host. Failed to download NSX components on host. Check the host connectivity and if "/tmp" folder has enough space" pointing to either a host connectivity issue or not enough space in the hosts /tmp directory.
  
  
• A similar error message stating "Failed to apply Component nsx-lcp-bundle(<version>)..." may alternatively be seen in NSX Manager like shown in this below image:
  
  The Resolve button will not be available for this error.

• The below entries may be encountered, indicating a host connectivity issue in the /var/log/proton/nsxapi.log on the NSX Manager. 
  

  <Time_Stamp> ERROR ActivityWorkerPool-1-11 Esx60SfdmManager 266533 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP26168"
   level="ERROR" subcomp="manager"] Install/Upgrade on ESX - Uploading LCP bundle to host ####c8bd-####-####-####-c87b22###### failed
  com.vmware.nsx.management.ovfops.exception.CommandExecutionFailed:    % Received % Xferd  Average Speed   Time    Time     Time  Current
  
  105M      0   359k  0:07:31  0:04:59  0:02:32  327k
  curl: (28) Operation timed out after 300000 milliseconds with 0 bytes received
  
  
  <Time_Stamp> ERROR ActivityWorkerPool-1-11 DeploymentUnitActivityInstall 266533 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP26050" level="ERROR" subcomp="manager"] Host prep failed for ####c8bd-####-####-####-c87b22######.
  com.vmware.nsx.management.service_fabric.sfdm.exception.HostInstallFailedException: null
  
  

• • In the above snippet, we see that the operation times out as the NSX LCP file is not copied onto the "/tmp" directory of the ESXi Host within 300,000 milliseconds (300 seconds ~ 5 minutes).
    
    
• For the hosts /tmp file system available space, run 'vdf -h' from an SSH session on the host. 
  
  
• The nsx-lcp-bundle size varies in size but can reach 175M. 
  
• 
  
  
• Network latency is observed between the NSX Manager and the vCenter:
  

  traceroute --mtu <Destination IP> -I
  traceroute to 10.x.x.36, 30 hops max, 65000 byte packets
   1 10.x.x.32 0.294 ms F=1500 0.299 ms 0.215 ms
   2 10-X-x-34 50.804 ms 50.655 ms 59.154 ms
   3 10.x.x.35  219.656 ms 220.023 ms 219.275 ms
   4 10-x-x-36  223.441 ms 221.812 ms 224.755 ms

Environment

VMware NSX-T DataCenter
VMware NSX

Cause

This issue can occur if the nsx-lcp-bundle file takes more than 5 minutes to transfer to the ESXi host, or if there is not enough free space in the hosts /tmp file system for the nsx-lcp-bundle file. 

Resolution

This is a condition that may occur in a VMware NSX environment.

• Before trying the workaround below, check the status of the REPO_SYNC on the NSX managers. Confirm that the REPO_SYNC is showing as SUCCESS on the NSX managers 

If any are showing as FAILED, click RESOLVE  to restart the resync 

 

Workaround

Manually copy the LCP file from the NSX Manager node from /repository into the /tmp folder of the ESXi host, which can be achieved via WinSCP.
Note: If REPO_SYNC is healthy, but latency is high and cannot be fixed; the manual workaround is the primary path forward.

1. Locate the LCP bundle on the NSX Manager: 
   

   /repository/<NSX-version>/HostComponents/<esx-version>/nsx-lcp-<nsx-version>-<esx-version>.zip
   
   i.e. /repository/4.2.0.0.0.24105817/HostComponents/esx80/nsx-lcp-4.2.0.0.0.24105819-esx80.zip

2. Use WinSCP or SCP to copy the bundle from the Manager to the ESXi host's /tmp directory.
   
   
3. Run the following command on the ESXi host: Review this command before running it.
   

   esxcli software component apply -n nsx-lcp-bundle  -d /tmp/nsx-lcp-<nsx-version>-<esx-version>.zip --no-sig-check

4. Return to the NSX Manager UI to confirm that the host preparation completes.