NSX-MGR node shows intermittent connectivity to transport nodes.

Products

• VMware NSX

Issue/Introduction

• NSX-MGR node has intermittent communication to transport nodes.
• In NSX-MGR UI, view the status of the ESXi hosts/cluster configured for NSX. It may be observed that the status changes from "UP" to "Degraded" periodically.
  
• Clicking on "view details" shows controller connectivity is "down".
  
• Checking the status of the NSX-MGR nodes shows connectivity alarms to few transport nodes. 
  

• Logging into NSX-MGR via SSH and using the "ping" command shows no ICMP response is received.
• ARP replies are not being delivered to ESXi uplink attached to NSX-MGR. 
• This can be verified by using the pktcap-uw command on ESXi to trace at the uplink the NSX-MGR is connected to.
• SSH to the ESXi host the NSX-MGR is operating on. Use the command 'esxtop' then press 'n' for networking to find the vmnic the NSX-MGR vm is using.
• Once you have the correct uplink, run the below command to see if ARP replies from the ESX mgmt. VMK are being received.
  Example:
  pktcap-uw -- uplink vmnic# -- capture UplinkRcvKernel -o - | tcpdump-uw -r - -ean arp |grep 'TargetIP'
*Target IP can be either host mgmt VMK or NSX-MGR mgmt IP.
  
  
• If you wish to verify if ARP requests are being sent out from a host uplink use the below command:
  pktcap-uw -- uplink vmnic# -- capture UplinkSndKernel -o - | tcpdump-uw -r - -ean arp |grep 'TargetIP'

Environment

VMware NSX 

Cause

• ARP reply from transport node is not being forwarded to NSX-MGR uplink.


Resolution

• Engage the physical switch owner to understand why the MAC is not being forwarded to the ESXi uplink/vmnic attached to the NSX-MGR vm.