Intermittent connection between NSX-v Host vsfwd agent and NSX manager

Products

• VMware NSX

Issue/Introduction

• ESXi host vsfwd agent intermittent connection to NSX manager on port 5671 . This issue persists even after agent restart and bus sync through API.
• In the vsfwd.log file, you see entries similar to:

<Date>T<Time>Z vsfwd: [INFO] Host <IP address>, Port 5671, Host Id host-<moid>, PeerExchange com.vmware.vshield.userworld_exchange, Request Q com.vmware.vshield.userworld_<uuid>_request_queue, ResponseQ com.vmware.vshield.userworld_<uuid>_response_queue, VHost vshield, ClientId <UUID>, VSM Exchange vsm_exchange
<Date>T<Time>Z vsfwd: [INFO] Opened NET Socket fd 9
<Date>T<Time>Z vsfwd: [INFO] Listening on 100007F:15002...
<Date>T<Time>Z vsfwd: [INFO] NET Connected to 7f000001:55592.
<Date>T<Time>Z vsfwd: [INFO] Registration received for AppType VNET AppId host-<moid>
<Date>T<Time>Z vsfwd: [INFO] Registered App of AppType VNET and AppId host-<moid> at index 5
<Date>T<Time>Z vsfwd: [INFO] Do not send app topic request before establishing connection to VSM
<Date>T<Time>Z vsfwd: [ERROR] Error While Processing Data from Client -1005
<Date>T<Time>Z vsfwd: [INFO] Removed App of AppType VNET and AppId host-<moid>
<Date>T<Time>Z vsfwd: [INFO] Re-read credentials to broker <IP address>:5671: Logging in: Input/output error
<Date>T<Time>Z vsfwd: [INFO] Ending NET CLIENT Listen Thread
host ID in the log is host-<moid>

• Host ID is different in vsfwd and NSX Manager DB.
  
  Domain.com> show cluster domain-<moid>
  Datacenter: DomainData
  Cluster: TestCluster
  No. Host Name Host Id Installation Status
  1 <Hostname> host-<moid> Ready
  2 <Hostname> host-<moid> Ready
  3 <Hostname> host-<moid> Ready

Environment

VMware NSX for vSphere

Cause

This issue occurs due to misconfiguration in host profile.

Resolution

To resolve this issue, deselect all UserVars.Rmq* variables under Advanced configuration option from host profile and reapply to the host profile.

# grep Rmq /etc/vmware/esx.conf

/advUserOptions/options[0030]/name = "RmqClientPeerName"

/advUserOptions/options[0031]/name = "RmqHostId"

/advUserOptions/options[0036]/name = "RmqClientResponseQueue"

/advUserOptions/options[0037]/name = "RmqClientExchange"

Note: These variables are host-specific.