NSX Host preparation process stuck at 23 % in vSphere Lifecycle Manager due to UUID mismatch

Products

• VMware NSX

Issue/Introduction

• The host preparation process within vSphere Lifecycle Manager (vLCM) fails at 23% due to a UUID mismatch between the host's local configuration and the NSX Manager.

• CLI validation with the command get node-uuid on the transport node post invoking nsxcli shows a mismatch between the Host's local UUID and the UUID in the NSX Manager UI (Fabric > Hosts):

  Example from Host :
  host-1> get node-uuid
  uuid: 123########
  From NSX manager UI :
  Fabric > hosts> select the issue host > click on 3 dots and copy the id to clipboard.
  UUID : fdc#####

• Connectivity between the Host and both the Controller and Manager is down.

• While searching the host on NSX elastic search we can see the stale segment ports are visible in NSX elastic search

Environment

VMware NSX 

Cause

• A discrepancy between the local host UUID and the NSX Manager UUID was caused by stale segment ports remaining connected to the NSX host.
• These stale ports triggered the UUID mismatch between the host and the NSX Manager.
• This discrepancy is causing connectivity issues with both the controller and the manager on the host.

Resolution

1. Place the Host in Maintenance Mode to prevent disruption.
2. Delete Stale Segment Ports via the NSX UI.
3. Perform Search Resync on all three NSX Managers using these commands with admin mode login:
   • >start search resync all
   • >start search resync telemetry
4. Move the Host from the Cluster level to the Data Center (DC) level to remove NSX.
5. Force Delete the Transport Node from the NSX UI if the host remains in an orphaned state.
6. Verify VIB Removal by running esxcli software vib list | grep -i nsx  on the Transport node. Confirm no output is returned.
7. Manual VIB Removal: If VIBs remain, execute nsxcli -c del nsx from the Host CLI.
8. Reboot the node to meet vLCM preparation requirements.
9. Re-add the Host to the cluster to restart NSX preparation.

Additional Information

• If the above errors are not there and UUID is same in local host and NSX manager please refer to the below kb's to troubleshoot the error with similar error code:
  
  
  • Preparing ESXi host with NSX fails with an error "Waiting for connection to Managers"
  • NSX Transport Nodes ( Edge/ESXi host) show "Down/Degraded" due to DNS issue
  • Preparing ESXi host with NSX fails with an error "Failed to install software on host. Failed to install software on host. Time out waiting for host to join NSX Manager"