NSX Host Preparation Error: "Failed to send the HostConfig message."... "Failed to send HostConfig RPC to MPA node" while moving a Host from one cluster to another

Products

• VMware NSX

Issue/Introduction

While trying to move a host from an NSX prepared cluster with security to another NSX cluster, or when adding a standalone host to an NSX cluster, NSX fails to install.

The transport node preparation shows a Failed status with the error: Failed to send the HostConfig message. [TN=TransportNode/########-####-####-####-###########]. Reason: Failed to send HostConfig RPC to MPA node:########-####-####-####-############.

The error is repeatable, appearing each time the host is moved out and back into the cluster, or when moved into the original cluster.

Environment

VMware NSX 4.x

Cause

NSX was not properly uninstalled from the host.

Resolution

Evacuate the host & remove the host from the cluster. 

Ensure the host is not visible in the UI: System > Fabric > Hosts > Clusters, also check Other Nodes, and Standalone.

If the Host is still found on one of those locations, choose to Remove NSX.

Once that has completed successfully, from the command line of the ESXi host:

• Check for the presence of NSX VIBs: net-dvs -l | grep com.vmware.nsx
• Uninstall NSX if the VIBs are still present: nsxcli -c del nsx
• Clean up switches with security preparation using the workaround in NSX Host preparation fails with an "Host configuration: Found security-only non-CVDS on the host" message (KB 378540) 
  • net-dvs -l | grep -E "^switch|com.vmware.nsx.*.enable|com.vmware.common.alias"
  • For each identified property/DVS:
    • net-dvs -u com.vmware.nsx.<property>.enabled -p hostPropList <dvsName>
    • net-dvs -u com.vmware.nsx.kcp.enable -p hostPropList <dvsName>

Re-add the affected host to the new cluster, and NSX preparation of the host should complete successfully.

Additional Information

To prevent this problem, follow the steps in: Uninstall NSX from a Managed Host in a vSphere Cluster