ESXi Host Transport Node's Tunnels Shows as Down in NSX UI

Products

• VMware NSX

Issue/Introduction

• The tunnels shows as down status in NSX UI for ESXi Transport Nodes as seen below:
• 
• 
• vmkpings between the hosts TEP does not work:
  • vmkping -S vxlan -s 1460 -I vmkX -c 100 <Remote-Host-TEP-IP>
• ESXi host TEPs belongs to different subnet.
• Packet Captures on respective TEP (vmk10 or vmk11) interfaces and associated uplink (vmnicX) performed as stated below:
  • vmk11: BFD traffic is egressing from the TEP vmk interface.

#pktcap-uw --vmk vmk11 --dir 2 -o - | tcpdump-uw -enr - | grep -i <remote TEP IP> | grep -i <Local TEP IP> 

• • vmnicX: BFD traffic is not reaching the ESXI Host uplink.

#pktcap-uw --uplink vmnicX --capture UplinkSndKernel -o - | tcpdump-uw -enr - | grep -i <remote TEP IP> | grep -i <Local TEP IP>

• The default gateway was not reachable for respective host's TEP (vmk10 or vmk11) :
  • vmkping -S vxlan -s 1460 -I vmkX -c 100 <Remote-Host-TEP-gateway-IP>

Environment

VMware NSX

Cause

• Packet captures on respective host's TEP (vmk10 or vmk11) while pinging default gateway through respective host's TEP (vmk10 or vmk11):
  • #pktcap-uw --uplink vmnicX --capture UplinkSndKernel,UplinkRcvKernel -o - | tcpdump-uw -enr - | grep -i <Remote-Host-TEP-gateway-IP>
• In the packet captures, ESXi host's uplink show the ARP request successfully egressing (leaving) the host but no ARP replies been received on the host's uplink.
• This suggests the packet is being dropped outside the ESXi host by a component on the physical network (e.g., a physical switch).

Resolution

Since the packet drop is identified as occurring outside the ESXi host on the physical network, the issue must be resolved by the physical network team.

1. Engage the physical network team.

2. Request the team to check the physical network configuration (e.g., VLAN settings, switch port configuration, spanning tree) relevant to the two ESXi hosts' uplinks.

3. Instruct the team to trace the packets in the physical network infrastructure to precisely identify the drop point and correct the misconfiguration.