vCenter across two different sites becomes unreachable in a stretched and federated NSX cluster

Products

• VMware NSX
• VMware Telco Cloud Platform

Issue/Introduction

• In an NSX federated stretched Tier-0 topology, the vCenter at the secondary site became unreachable.
• The vCenter could not reach its local Edge Nodes and Edge-to-Edge communication between the Secondary site and Primary site was failing.
• Both vCenters successfully ping their shared distributed gateway, indicating that the issue is isolated to the overlay transport network rather than local distributed routing.
• Packet captures taken at the Source ESXi host uplinks revealed that Geneve-encapsulated ICMP echo requests were successfully sent from the source Edge node but received no replies.
• Subsequent packet captures at the destination ESXi host demonstrated that the physical gateway was broadcasting ARP requests for the destination RTEP IP, but the destination NSX Edge failed to respond to these requests. 

Environment

NSX 3.2.2 
TCP 2.7 and above

Cause

• The root cause is a chronic ARP resolution failure at the datapath layer between RTEP BGP peers with the default gateway or next-hop IP.
• While the control plane correctly installs static routes based on configuration, it does not actively track L2 (ARP) reachability for those next-hops.
• If the upstream physical gateway fails to respond to ARP solicitations, the Edge datapath enters an infinite loop of incomp -> failed states (visible in datapathd logs). This creates a silent blackhole where the control plane believes the path is valid, but the data plane cannot encapsulate or forward packets.

Resolution

To resolve this issue, the underlying network layer connectivity to the next-hop gateway must be restored.
Work with the network administration team to:

• Verify the status of the physical upstream router/gateway corresponding to the failing IP.
• Ensure the upstream device is not dropping ARP requests or configured with security policies that prevent ARP replies to the NSX Edge interfaces.
• Check for VLAN tagging mismatches or physical link failures between the Edge and the Top-of-Rack (ToR) switch.
• Implement proactive monitoring of the Edge ARP table. Once the upstream gateway begins responding to ARP requests, the BGP peers will automatically transition to an Established state.

If the issue persists after verifying upstream network connectivity, contact Broadcom by referring KB Broadcom Support