Failed to install/uninstall NSX on Host in vLCM enabled cluster with error "Failed to send HostConfig RPC to MPA"

Products

• VMware NSX

Issue/Introduction

• vLCM configured ESXi Hosts fail to install/uninstall NSX with "Install Failed" status in NSX UI and below error:
  
  "Host configuration: Failed to send the HostConfig message.
[TN=TransportNode/<uuid>]. Reason: Failed to send HostConfig RPC to MPA TN:<uuid>. Error: Unable to reach client <tn-uuid>, application SwitchingVertical."
  
  
• If installing NSX, it may fail at 48% at step: "Waiting for Connection to Managers"
• If uninstalling NSX, Hosts may go into an "Orphaned" state.

Environment

VMware NSX

Cause

This error message typically indicates a communication breakdown between the NSX Manager and the NSX agents (specifically the nsx-proxy or nsx-opsagent) running on the ESXi host. When vLCM (vSphere Lifecycle Manager) attempts to sync or remediate, it fails because it cannot "hand off" the configuration message to the host's management plane.

The "SwitchingVertical" component refers to the internal NSX module responsible for logical switching and host-side networking configuration.

Resolution

Resolution / Workaround:

To successfully uninstall and reinstall NSX on an affected ESXi Host within a vLCM-enabled environment, follow the steps below:

1. Place the Host in Maintenance Mode (MM):
   Migrate all workloads off the affected host and place it into maintenance mode.

2. Move Host to DataCenter Level in VC:
   Migrate Host outside of NSX prepared cluster to the DC level.

3. Remove NSX from Host via UI:

   1. Navigate to System > Fabric > Hosts > Standalone/Other Nodes.

   2. Select the affected host and click Delete NSX.

   3. If the host status changes to Orphaned, reselect the host and click Delete NSX > Force Delete NSX.

4. Verify Host Status:
   Confirm the host now shows as Not Configured in the NSX Manager UI.

5. Add Host Back to Cluster:
   After Status shows as Not Configured as stated in Step 4, add host back to cluster to reinitiate NSX configuration on this host.

6. Proceed to Steps 7-12 if Step 1-5 still resulted in Failed Install:
   Proceed with steps below if the installation still fails with [TN=TransportNode/<uuid>]. Reason: Failed to send HostConfig RPC to MPA TN:<uuid>. Error: Unable to reach client <tn-uuid>, application SwitchingVertical."

7. Confirm VIB Removal via CLI:
   SSH into the host and run the following to check for remaining NSX VIBs:esxcli software vib list | grep nsx

8. Manually Remove VIBs if Still Present:

   1. Stop the nsx-proxy service: /etc/init.d/nsx-proxy stop

   2. Remove NSX components: nsxcli -c del nsx

9. Verify VIBs are Fully Removed:
   Run the esxcli command again to confirm all NSX VIBs have been removed:esxcli software vib list | grep nsx

10. Reboot (for vLCM-managed Hosts):
    If the host is managed by vLCM, a reboot is required after the VIBs are removed using the nsxcli -c del nsx method above. Wait for the VIBs to be cleared before initiating the reboot.

11. Migrate Host into Cluster:
    After reboot and confirmation that VIBs are removed, migrate the host back into the vLCM-enabled NSX cluster. NSX installation should reinitiate automatically.

12. Post-Validation:
    Once installation completes successfully:

    1. Exit Maintenance Mode.

    2. Migrate a few VMs to the host and verify that Geneve tunnels and other services come up as expected.

NOTE: If the above symptoms are seen while uninstalling NSX from ESXi host, the same steps can be followed until Step 10 to uninstall NSX from the ESXi host.