Message Bus is down on ESXi host after issue with host preparation

Products

• VMware NSX

Issue/Introduction

• Message Bus in Communication Channel Health is down on the host.
• ESXi host vsfwd logs show entries similar to:
  
  

  2018-02-14T14:07:29Z vsfwd: [INFO] Re-read credentials to broker ##.##.##.##:5671: Logging in: Input/output error
  2018-02-14T14:07:29Z vsfwd: [ERROR] Bad pw file length: 0
  2018-02-14T14:07:29Z vsfwd: [ERROR] Failed to get credentials to broker ##.##.##.##:5671: -1

 

• On the NSX Manager, you may see the below entries in the [email protected]

=ERROR REPORT==== 14-Feb-2018::14:07:21 ===
Error on AMQP connection <0.17646.4> (127.0.0.1:56160 -> 127.0.0.1:5672, state: starting):
PLAIN login refused: user 'uw-host-##' - invalid credentials

• ESXi host cannot establish the socket to the NSX Manager when runningesxcli network ip connection list | grep 5671 , similar to:
  
  

  #esxcli network ip connection list | grep 5671
  tcp         0       0  ##.##.##.##:19158    ##.##.##.##:5671   TIME_WAIT           0

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX Data Center for vSphere 6.4.x

 

Cause

If the host failed to connect to the NSX Manager within 2 hours after host preparation completed, its password expires.
This typically happens after fixing an issue preventing the ESXi host from connecting to the NSX Manager (after host prep).

For example:

• vShield-Stateful-Firewall could not start.
• The socket was blocked by a physical firewall.
• The traffic was filtered by the ESXi firewall (maybe because the vSFW-UW ruleset was disabled).

Resolution

This is a condition that may occur in a VMware NSX Data Center for vSphere 6.4.x environment.

 

Workaround

To resolve the occurrence, reset the communication between NSX Manager and the host (or cluster) via REST API call.

For example:

POST https://<NSXMGR_IP>/api/2.0/nwfabric/configure?action=synchronize

<nwFabricFeatureConfig>
    <featureId>com.vmware.vshield.vsm.messagingInfra</featureId>
    <resourceConfig>
        <resourceId>host-##</resourceId>
    </resourceConfig>
</nwFabricFeatureConfig>