Increase in memory utilization on a VMware SD-WAN Edge causing service restart

Products

• VMware SD-WAN by VeloCloud

Issue/Introduction

Symptoms:

Memory usage of the VCE increases continuously and generates memory usage warnings such as "Process edged memory usage exceeds % threshold"
Warning may appear every few hours in VCO Events
More commonly observed on entry level Edge models (for example, the 510, 610, or 620) which have smaller amounts of Edge memory, but over a long enough period every Edge model could reach a critical memory level (60% memory utilization for longer than 90 seconds) and restart.

Environment

VMware SD-WAN by VeloCloud
VMware SD-WAN

Cause

This issue (Issue 89873) occurs when UDP flows with unique IP address and ports are processed at a high rate on the Edge. Flow creation is handled asynchronously on the Edge and when multiple packets of a same flow are enqueued to the flow creation service, the flow objects are leaked and result in an Edge memory leak.  

Resolution

This issue (89873) is resolved in the following Edge software versions or later:

4.3.2 (R432-20221115-GA)

4.5.1 (R451-20220701-GA) 

5.0.1.0 (R5010-20220729-GA) 

 

For information on how to upgrade please check the following article: https://kb.vmware.com/s/article/67152

Workaround:

When memory utilization reaches 40% and the Orchestrator records a Memory Warning Event, the recommendation is to schedule an Edge Service Restart in a maintenance window to clear the memory and ensure minimal customer impact.

Additional Information

Impact/Risks:
Workaround involves an Edge service restart which might cause a brief service interruption