VMware VeloCloud SD-WAN general troubleshooting of EDGE_CONGESTED event

Products

• VMware VeloCloud SD-WAN

Issue/Introduction

After upgrading to 5.1 or newer versions, customer may observe SD-WAN edge (especially hubs that provide service) generating EDGE_CONGESTED event followed with EDGE_STABLE event . They are new events introduced on 5.1.

Environment

SD-WAN edge 5.1 and subsequent versions

Cause

This event is generated when scheduler drops larger than 50. As a hub typically has many spokes, if hub has scheduler drop towards any one of spokes, hub generates EDGE_CONGESTED event.

Resolution

Below are few things customer can check before digging deep into scheduler drop:

1. Check if any spoke has very low overall download bandwidth. For example, hub has large overall upload bandwidth while a spoke has very low overall download bandwidth, when the low bw sites downloads bigger size files, the hub is not able to push the traffic faster on the tunnel hence causing momentary drops due to sched buffer exhaustion.

2. Check if any biz policy on the spoke side that has rate-limit enabled. The reason is same as above.

3. Check link throughput of hub around EDGE_CONGESTED event's timestamp. When link is congested, hub usually transmit high priority flows, thus flows with lower priority is buffered. After buffer is exhausted, scheduler drop happens. 

If there is no obvious problematic sites identified from above three steps, customer needs to dig deeper to QoS_net and QoS_link to find the scheduler drop.Please see below flow chart to troubleshoot Scheduler drops.