NSX host preparation stalls at 15% during vLCM remediation - VMware NSX

Products

• VMware NSX

Issue/Introduction

• vSphere Lifecycle Manager (vLCM) is a centralized tool within vSphere that manages the lifecycle of ESXi hosts, clusters, and virtual machines. 
  It is applied at the cluster level. 
• When adding an ESXi host to a vSphere Lifecycle Manager (vLCM) enabled cluster, the NSX Transport Node preparation stalls at 15%.
• vLCM will ensure that all hosts of the cluster have the same ESX and NSX versions applied to all hosts of the cluster.
• This requires that vLCM have 2 images.  One for NSX and the other for ESX. 
  The ESXi image can be obtained from the image directory located within VMware vCenter.
• The NSX image is automatically uploaded to VMware vCenter when the NSX cluster preparation commences.  
  vLCM obtains the NSX image only when a cluster is prepared for NSX networking.
• If during vLCM's validation either image is not the same as the vLCM image it will stop, and message that the host is not in compliance will be thrown.
  
  This article addresses a situation where a host added to the vLCM enabled cluster will not complete the install.  In the NSX UI it may show an NSX Configuration status of Uninstall Skipped or stall at 15%(vLCM remediation initiated) of installing NSX on the host.
  
  
  

 

Environment

VMware NSX

vSphere Lifecycle Manager (vLCM) enabled clusters

Cause

• The cause of this issue is some mismatch detected by vLCM, when comparing what image the host has currently with what vLCM has for a compliant host image.
• A compliance mismatch exists between the host's current ESXi image and the desired cluster image managed by vLCM. This often creates a circular dependency where NSX waits for vLCM remediation, while vLCM fails due to NSX health check pre-requisites.
• This sometimes can be resolved by pulling the host out of the cluster.  Then ensuring that NSX VIBs are not installed.  Finally, re-add the host to the cluster.
• If it still fails to complete the install of NSX at 15% ( vLCM host validation) this is not an issue with NSX.
• There is a discrepancy between vCenter vLCM and the ESXi host that vLCM needs to have investigated. 

Resolution

To resolve this issue, perform a manual cleanup of the host state to allow vLCM to synchronize correctly:

1. 1. Place the affected ESXi host into Maintenance Mode.
   2. Move the host out of the vLCM-enabled cluster to the Data Center level in vCenter.
   3. Verify the host does not have residual NSX VIBs by running the following command via SSH: esxcli software vib list | grep nsx
   4. If VIBs are present, remove them manually and reboot the host.
   5. Move the host back into the vLCM cluster.
   6. Trigger a manual Remediate task in the vLCM cluster view for the host.
   7. Verify the host shows as Compliant before checking the NSX preparation status.

• If NSX is successfully installed, then perform the same process on the remaining hosts.
• This is the workaround when a vLCM cluster will not validate a host and stalls at 15%, or NSX configuration state is reported as 'Uninstall Skipped'.

Additional Information

If the issue persists, refer to Contact Broadcom Support for further assistance.