Message Bus is down on ESXi host after issue with host preparation

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• Message Bus in Communication Channel Health is down on the host.
• ESXi host vsfwd logs are similar to:
  
  2018-02-14T14:07:29Z vsfwd: [INFO] Re-read credentials to broker 10.202.100.39:5671: Logging in: Input/output error
  2018-02-14T14:07:29Z vsfwd: [ERROR] Bad pw file length: 0
  2018-02-14T14:07:29Z vsfwd: [ERROR] Failed to get credentials to broker 10.202.100.39:5671: -1

 

• On the NSX Manager, you may see the below entries in the [email protected]

=ERROR REPORT==== 14-Feb-2018::14:07:21 ===
Error on AMQP connection <0.17646.4> (127.0.0.1:56160 -> 127.0.0.1:5672, state: starting):
PLAIN login refused: user 'uw-host-16' - invalid credentials

• ESXi host cannot establish the socket to NSX Manager when running the esxcli network ip connection list |grep 5671 similar to:
  
  [root@esxi-0835:~] esxcli network ip connection list | grep 5671
  tcp         0       0  10.202.8.46:19158    10.202.100.39:5671   TIME_WAIT           0
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

If the host failed to connect to the NSX Manager within 2 hours after host preparation completed, its password expires.
This typically happens after fixing an issue preventing the ESXi host from connecting to the NSX Manager (after host prep). For example:

• vShield-Stateful-Firewall could not start.
• The socket was blocked by a physical firewall.
• The traffic was filtered by the ESXi firewall (maybe because the vSFW-UW ruleset was disabled).

Resolution

To resolve this issue, reset the communication between NSX Manager and the host (or cluster) via REST API call.

For example:

POST https://<NSXMGR_IP>/api/2.0/nwfabric/configure?action=synchronize

<nwFabricFeatureConfig>
    <featureId>com.vmware.vshield.vsm.messagingInfra</featureId>
    <resourceConfig>
        <resourceId>host-15243</resourceId>
    </resourceConfig>
</nwFabricFeatureConfig>