ESXi hosts are in "Failed" status with errors "Failed to send the HostConfig message"

Products

• VMware NSX

Issue/Introduction

ESXi hosts are in "Failed" status on the NSX UI with below error:

 [TN=TransportNode/<uuid>]. Reason: Failed to send HostConfig RPC to MPA TN:<uuid>. Error: Unable to reach client <tn-uuid>, application SwitchingVertical."

Environment

VMware NSX

VMware NSX-T Datacenter

Cause

The NSX Manager cluster is degraded because the messaging-manager service is down on one of the nodes

Resolution

1. Verify from NSX UI: 

System > Appliances, selecting View Details for an NSX Manager node shows that the messaging-manager service is down.

2.Verify from NSX CLI: Running command: 

NSX-MGR> get service messaging-manager
Service name:      messaging-manager
Service state:        stopped <----

3. Restart the messaging manager service by running command:  "restart service messaging-manager" from admin shell of NSX manager.

NSX-MGR> get service messaging-manager
Service name:      messaging-manager
Service state:     running <-----

4. Verify NSX cluster is stable and healthy by running command > "get cluster status"

5. From the NSX UI, select the failed host, then click Configure NSX and proceed with Next followed by Finish.