VMware SD-WAN by VeloCloud Hub Edge reports an "Edge Tunnel CAP warning" event.

Products

• VMware SD-WAN by VeloCloud

Issue/Introduction

In the VMware SD-WAN Hub and Spoke architecture, the Hub Edge is commonly used as a transit point between the customer's SD-WAN and the non-SD-WAN domains and maintains static tunnels with the Spokes assigned to it.

For an Edge that has reached or exceeded the number of tunnels allowed for that Edge platform, there are three immediate implications:

 

1. Dynamic tunnels are consistently capped.
2. Outbound static tunnels (Edge to Gateway, Spoke to Hub) are always permitted due to their minimal number, and disallowing them would result in a complete outage.
3. Inbound static tunnels on Hub or Gateway are consistently capped as the quantity is unpredictable. Spokes should have an alternative connection, and if not, it is preferable for that spoke to be inactive rather than risking disruptions for all other spokes connected to the Hub.

There is a significant performance risk once the Hub Edge has exceeded its tunnel capacity.  A single software thread monitors all tunnels for a particular Edge.  If the Edge has built a number of tunnels in excess of its capacity, the tunnel monitoring thread can take too long to check all the tunnels.  This results in tunnels timing out and being torn down and then rebuilt with great frequency. When the static tunnels are flapping like this, users at many of the Spoke sites connected to this Hub Edge will report major issues for any activity that routes through that Hub (e.g. reaching the Internet if backhauling is configured). 

Environment

VMware SD-WAN by VeloCloud

Resolution

In the VMware SD-WAN Hub and Spoke topology, if the Hub Edge reports an "Edge Tunnel CAP Warning" to the VMware SD-WAN Orchestrator, this event message indicates that the Edge hardware has reached its maximum tunnel capacity.

If these messages are consistently posted, the customer has several options to address this issue:

1. Replace the existing Hub Edge with a larger capacity Edge model.
2. Add another Hub Edge and revise the network topology so that each Hub Edge is responsible for a portion of the total number of Spoke Edge's such that each Hub Edge handles only as many tunnels to Spoke Edges as it is specified to support..
3. Turn the Hub site into a Cluster Hub Site and add another Edge of the same model. Creating a Cluster and adding another Hub Edge doubles the site's tunnel capacity while not changing the network topology.
4. If Dynamic Branch-to-Branch VPN is enabled, consider disabling that feature for the Hub Edge.

Regarding an Edge's specified tunnel capacity, please consult the SD-WAN Performance and Scale
Datasheet, available here: https://www.vmware.com/content/dam/digitalmarketing/velocloud/en/documents/SD-WAN-Edge-VeloCloud-DS.pdf

Adding another Edge, or upgrading to a more powerful Edge model is handled through your VMware SD-WAN by VeloCloud Sales representative.

If the Hub Edge should have sufficient capabilities for the current deployment but is getting these "Edge Tunnel CAP warning" messages, please capture a diagnostic bundle and reach out to VMware SD-WAN  by VeloCloud Support through one of the methods outlined here: https://kb.vmware.com/s/article/53907