SD-WAN Orchestrator down causes production issue

Products

• VMware VeloCloud SD-WAN

Issue/Introduction

During SD-WAN Orchestrator downtime, end customer may find their production is impacted especially after SD-WAN Edge rebooting or BGP/OSPF flapping. The issue is gone right after SD-WAN Orchestrator back online and Edge<->Orchestrator heartbeat is restored.

Environment

All supported version of VMware by Broadcom SD-WAN

Cause

In Dynamic Routing, SD-WAN Edge learns routes from adjacent routers through OSPF or BGP. The SD-WAN Orchestrator maintains all the dynamically learned routes in a global routing table called the Overlay Flow Control (OFC).

Once a route is learned by a SD-WAN Edge, it will send the route to Orchestrator which will put the route into OFC and determine if the route should be advertised into overlay.

During Orchestrator downtime, if a SD-WAN Edge reboots, or BGP/OSPF flaps where Edge need to send all re-learned routes to Orchestrator to get a green light for advertising those routes, the process will stuck as VCO is down at the moment. Consequently other SD-WAN Edges within the customer enterprise are not able to learn those routes and production is compromised.

Below is a summary for all kinds of scenarios:

BGP/OSPF enabled	Edge rebooting or BGP/OSPF flapping？	Production affected
Yes	Yes	Yes
Yes	No	No
No（L2 or static routing）	Yes	No
No（L2 or static routing）	No	No

Resolution

Enable Distributed Cost Calculation Overview (DCC) for the customer enterprise. DCC is a feature that leverages the SD-WAN Edges and Gateways for route preference calculation instead of relying on the SD-WAN Orchestrator. The Edge and Gateway each insert the routes instantly upon learning them and then convey these preferences to the Orchestrator. DCC resolves an issue seen in large scale deployments where the Orchestrator is down or could not be reached by an Edge or Gateway to receive updated routing preferences.