Getting the message "Dropped due to ARP failure" when testing routing in Traceflow

Products

• VMware NSX

Issue/Introduction

• You are testing North-South connectivity and Traceflow shows "Dropped due to ARP failure," despite BGP peering being healthy and routes present in the forwarding table.
  
  
• Traceflow analysis reveals packets are "Dropped due to ARP failure" before reaching the transit-bp interface on the ESXi. This indicates a drop within the ESXi's prior reaching the edge node/s.
  
  Below is a sample screenshot from Traceflow:
  
  
  
  
• ESXi logs show vmk10 and/or vmk11 (tunnel endpoints) are configured for DHCP but are assigned IP addresses from range 169.254.x.x (APIPA) due to a lack of DHCPOFFERs.
  
  Note: 169.254.x.x can be configured to be used as TEP IP's, however in this case that IP range is not being used purposefully and is not expected
  
  Below are the sample logs FYR:
  <Timestamp> In(30) dhclient-uw[20####1]: No DHCPOFFERS received.
  <Timestamp> In(30) dhclient-uw[20####1]: No DHCPOFFERS received.
  
  
• NSX exporter logs show "Overlay tunnel summary -- up: 0, down: 0, unknown: 0" and "EVPN tunnel summary -- up: 0, down: 0, unknown: 0" on the ESXi, confirming no tunnels are established.

Environment

VMware NSX
VMware NSX-T Datacenter

Cause

The network unreachability stems from a failure in establishing tunnels between the ESXi hosts and the Edge nodes.
This is directly caused by the TEP vmk interfaces (vmk10 and vmk11) failing to obtain valid IP addresses via DHCP, instead auto-assigning APIPA addresses, thus preventing proper tunnel initialization.

Resolution

If the APIPA addresses are unexpected and are not configured purposefully, please investigate and resolve the DHCP service availability/reachability for vmk10 and vmk11 on the ESXi hosts.

This includes:

• Verifying DHCP server reachability from the hosts.
• Ensuring the IP pool has sufficient free IP addresses for these vmkernel interfaces.
  
  Once vmk10 and vmk11 receive valid IP addresses, the NSX tunnels should establish, restoring connectivity between ESXi TEPs and edge TEPs.

Note: If the issue persists after resolving TEP connectivity issues, feel free to open a case with Broadcom Support Team for further troubleshooting