Dynamic Branch to Branch tunnels fail to establish after a link flap.
Article ID: 323722
Updated On: 12-05-2024
Products
Issue/Introduction
Understand the behavior for Edges with multiple links and Dynamic Branch to Branch tunnels.
Symptoms:
-
VMware SD-WAN Edge has two active ISP links connected. In this example, the links are connected on the GE4 and GE5 interfaces.
-
Dynamic Branch to Branch tunneling is enabled in the configuration.
-
Dynamic Branch to Branch tunnels are established from both links to another Velocloud Edge. Meaning that a tunnel is formed from both GE4 and GE5.
-
One link flaps (Due to ISP issues, problem in the configuration, physical issues, etc) while the other link stays stable.
-
When the flapped link comes back, the dynamic tunnel is not reestablished.
Environment
Cause
The expected behavior when the circuit using the GE5 interface goes down and comes back up is for the Edge to rebuild all the static tunnels towards the VMware SD-WAN Gateways & Hubs. Meaning that the Edge already had a static tunnel established using it's GE4 link and after GE5 comes back up the, Edge should immediately establish another tunnel towards the Gateways using the GE5 link.
However, it is not mandatory for an Edge to establish a dynamic branch to branch over GE5 immediately if another path already exists between the two Edges. Dynamic Branch to Branch tunnels are temporary tunnels established between branches when there is live traffic between those two Edges.
Resolution
When an Edge needs to establish a dynamic tunnel with another Edge for Branch-to-Branch VPN traffic, it tries to establish the paths through all the available WAN links.
But when GE5 goes down and comes back up, because a path already exists between the two Edges on GE4, Dynamic Branch to Branch path or tunnel establishment does not go into effect for GE5 and hence the Edge does not initiate another path from the GE5 link after it flapped.
Should the GE4 link also go down and come back up, then the Edge would attempt to build paths via both WAN interfaces. This is the expected behavior.
When there are no existing Dynamic Branch to Branch towards a destination, the edge attempts to create VCMP paths over all the available WAN interfaces. But when one or more paths already exist towards the destination edge, the existing path(s) are used. Keep in mind that these are "dynamic" tunnels and not meant to stay up indefinitely. Dynamic tunnels remain up as long as they are needed and are torn down when there is no traffic flow between the Edges.
To check which tunnels are established from the Edge to the Gateways, please go to Remote Diagnostics > 'Edge Name' > List Paths:
Consider same scenario, GE4 and GE5 establish dynamic tunnel with another SD-WAN edge, when GE5 goes down, tunnel to peer SD-WAN edge tears down.
If there is no traffic flow between the SD-WAN edges, peer's WAN IP will be removed after very short time:
2024-08-07T06:52:28.404 1 [LINKMGR] vc_link_tunnel_init_one:452 Removing dead current destination:<WAN IP of peer edge> from GE5 link destinations list.
if there is traffic flow between the SD-WAN edges, SD-WAN edge will try to re-establish the dynamic tunnel. SD-WAN edge sends 20*20 MP_INIT_REQ packets in one round, interval between each MP_INIT_REQ packet is 700ms .
If twenty rounds fail to establish a tunnel, there will be a pause of 300 seconds. After twenty pauses, the WAN IP address will be removed from the destination list, and no tunnel will be established to that address thereafter.
Additional Information
If the remaining dynamic tunnels are not getting torn down due to ongoing traffic, and if you want to force the recreation of all the tunnels, the options are:
a) Perform a service restart on the initiator edge so that the tunnels are recreated.
b) Engage VeloCloud support, so that they can delete the existing dynamic tunnel towards the specific spoke using from cli. Refer to this kb while engaging the support.
Feedback
