Connectivity issue observed for single ESXi host with NSX manager in vLCM managed cluster

Products

• VMware vSphere ESXi
• VMware NSX

Issue/Introduction

On a vLCM managed ESXi cluster, connectivity issue was observed for a single ESXI host with NSX manager while preparing the host for NSX

The NSX Manager UI reports the following validation error:

Validation Errors  
601: The object HostTransportNode//infra/sites/default/enforcement-points/default/host-transport-nodes/<Host-uuid:host-id> is already present in the system.

Environment

ESXi 8.0U3 24784735

NSX 4.2.3.1 

Cause

The vendor addon version updated on the vLCM cluster differed with the vendor version updated on that specific ESXi host which was failing 

On vcenter, the "Image" Section of "Updates" tab for the cluster shows that HPE add on image on this single ESXi host is not compatible with that present in the cluster

Resolution

• To get rid of the validation error 601 clear the orphaned host using below API 

curl -k -u 'admin:<password>' -X DELETE "https://<nsx-mgr-ip>/policy/api/v1/infra/sites/default/enforcement-points/default/host-transport-nodes/<host-uuid>"

• Update all the ESXi hosts by installing the latest addon from Broadcom Support Portal and check if ESXi host comes up with NSX vibs installed successfully.

Additional Information

After the cluster is remediated successfully with latest vendor addon and still if the ESXi host remains in NSX Install failed or disconnected state , perform the below steps 

• Set the host to MM, and remove the failed ESXi host out of cluster, ensure the host moves to other nodes in NSX GUI.
• Try to remove NSX from NSX GUI. 
• If removing NSX vibs from GUI is not successful, manually remove vibs using command (by logging to ESXi via SSH) >> del nsx 
• After del nsx command execution, check vibs are removed using command (by logging to ESXi via SSH)  >> esxcli software vib list | grep -i nsx 
• Reboot the ESXI host as the cluster is managed by vLCM.
• Add the host back to the cluster and after the host comes up and monitor the status of NSX installation on NSX GUI and from ESXI host by checking /var/log/esxupdate.log
• After successful vibs installation, check vmkernel adapters for NSX are installed and the host is UP and operational with NSX