Northbound Communication Failures for VMs in NSX Federation

Products

• VMware NSX

Issue/Introduction

Loss of connectivity from VMs to external networks.
NSX Traceflow results indicate traffic is "Dropped due to ARP failure" at the Edge node or Tier-1 Gateway.

Remote Tunnel Endpoint (RTEP) to RTEP ping tests fail. 
Packet captures confirmed that the ARP broadcast requests sent by the local Edge node were never received at the destination RTEP host uplink. 

Environment

VMware NSX Federation

Cause

The physical underlay network is dropping GENEVE-encapsulated packets (UDP port 6081) 
Incorrect MTU settings on physical switches (GENEVE requires a minimum MTU of 1600).

Resolution

Perform the following steps to isolate the underlay issue:

Connectivity test between RTEP to RTEP: 

1. Identify the RTEP VRF: https://knowledge.broadcom.com/external/article/369203 

1. Log in to the NSX Edge CLI.
2. Run get logical-routers.
3. Locate the entry for REMOTE_TUNNEL_VRF and note its VRF ID (e.g., 1) 

2. Enter the VRF and Verify the Interface: https://knowledge.broadcom.com/external/article/369203 

1. Run vrf <VRF_ID> (e.g., vrf 1).
2. Run get interfaces to confirm the remote-tunnel-endpoint interface is Up and has the correct IP address.

3. Perform Ping and Path Validation: https://knowledge.broadcom.com/external/article/369203 

• Standard Ping: Run ping <Remote_RTEP_IP> to check basic Layer 3 reachability. 
• MTU Validation: To verify MTU consistency across the physical underlay, use a large packet size with the "do not fragment" bit set: ping <Remote_RTEP_IP> repeat 5 size 1550 dfbit ENABLE  

 Take packet captures on REMOTE_TUNNEL_VRF uplink interface to verify that ARP, ICMP, BGP packets are received from underlay

1. Invoke the NSX CLI command  get logical-routers on the affected edge node.
   
2. Switch to REMOTE_TUNNEL_VRF context.
   
3.  Invoke the NSX CLI command get interfaces


   
   Note: Copy the uplink interface UUID and use that to capture packets on this interface
4. Invoke an NSX CLI command similar to the following to start the packet capture: start capture interface {interface_uuid} direction dual
   
   
5. For copying over the packet captures to a file:
   
   set capture session 1 interface {interface_uuid} direction dual
set capture session 1 file file_name.pcap


6. Capture to file initiated, enter Ctrl-C to terminate.
7. Issue an NSX CLI command similar to the following to copy the .pcap file to a remote system: copy file file_name.pcap <url>
   Note: <url> is a remote file url (e.g. scp://username@ip_address/filepath/filename)