Traffic from virtual machines fails to reach external network, dropping at NSX Tier-1 Gateway with ARP failure

Products

• VMware NSX

Issue/Introduction

• Traffic originating from virtual machines (such as AVI Service Engine VMs) is unable to reach physical network destinations (such as an external AVI Controller). 
• When running an NSX Traceflow, packets drop at the Tier-1 Gateway with the following error: Dropped due to ARP failure.

• Note in the Traceflow output the Physical Hop Count starts at 0, and the last captured point is hop 0. This means the packet stopped on the same host it started.
• Sourced pings from the segment gateways fail.
• In the NSX GUI at System → Fabric -→Hosts, affected hosts show the GENEVE overlay tunnels between the ESXi host and the Tier-1 Edge nodes are down.

Environment

VMware NSX

VMware NSX-T Datacenter

Cause

The GENEVE tunnels between the Host Transport Node and the Edge nodes fail to establish due to an incorrect IP configuration in the Host Tunnel Endpoint (TEP) IP pool. The Host TEPs are assigned IPs from a different subnet than the Edge TEPs. If the specific underlay network requires the Host and Edge TEPs to reside on the same L2 subnet, having them on different subnets without the required underlay routing prevents the tunnels from establishing. Consequently, VM-sourced traffic requiring overlay transport to the Edge for northbound routing is dropped.

Resolution

 

• Create a new TEP IP pool with the correct subnet range to match the Edge TEP subnet.

• Apply this newly created IP pool to the Host Transport Node (TN) profile.

• Verify that the host receives a new TEP IP from the correct subnet.

• Verify that the overlay tunnels establish successfully and bidirectional network connectivity is restored.

Note: If the issue persists after resolving TEP connectivity issues, feel free to open a case with Broadcom Support Team for further troubleshooting