Message Bus is down on ESXi host after issue with host preparation
search cancel

Message Bus is down on ESXi host after issue with host preparation

book

Article ID: 312595

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms:
  • Message Bus in Communication Channel Health is down on the host.
  • ESXi host vsfwd logs are similar to:

    2018-02-14T14:07:29Z vsfwd: [INFO] Re-read credentials to broker 10.202.100.39:5671: Logging in: Input/output error
    2018-02-14T14:07:29Z vsfwd: [ERROR] Bad pw file length: 0
    2018-02-14T14:07:29Z vsfwd: [ERROR] Failed to get credentials to broker 10.202.100.39:5671: -1
 
=ERROR REPORT==== 14-Feb-2018::14:07:21 ===
Error on AMQP connection <0.17646.4> (127.0.0.1:56160 -> 127.0.0.1:5672, state: starting):
PLAIN login refused: user 'uw-host-16' - invalid credentials

  • ESXi host cannot establish the socket to NSX Manager when running the esxcli network ip connection list |grep 5671 similar to:

    [root@esxi-0835:~] esxcli network ip connection list | grep 5671
    tcp         0       0  10.202.8.46:19158    10.202.100.39:5671   TIME_WAIT           0


    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Cause

If the host failed to connect to the NSX Manager within 2 hours after host preparation completed, its password expires.
This typically happens after fixing an issue preventing the ESXi host from connecting to the NSX Manager (after host prep). For example:
  • vShield-Stateful-Firewall could not start.
  • The socket was blocked by a physical firewall.
  • The traffic was filtered by the ESXi firewall (maybe because the vSFW-UW ruleset was disabled).

Resolution

To resolve this issue, reset the communication between NSX Manager and the host (or cluster) via REST API call.

For example:

POST https://<NSXMGR_IP>/api/2.0/nwfabric/configure?action=synchronize

<nwFabricFeatureConfig>
    <featureId>com.vmware.vshield.vsm.messagingInfra</featureId>
    <resourceConfig>
        <resourceId>host-15243</resourceId>
    </resourceConfig>
</nwFabricFeatureConfig>