Edge nodes are unable to communicate with other site edge nodes via the RTEP network.
VMware NSX-T Data Center with NSX Federation
NSX Federation on VMware NSX with NSX Federation
This is an NSX control plane issue, there is a discrepancy between CCP L3 data and Pigeon data. This issue can be triggered under certain RTEP configuration operations which cause edge RTEP interfaces to have no default route configured.
This is a known issue that will be resolved in a later version of NSX.
Workaround
Restarting the control plane on 3 manager nodes will resolve the issue:
"admin"
userrestart service controller
The issue can be verified by entering RTEP VRF, running "get forwarding"
and then checking the output.