Aria Operations for Networks shows Processing Grid Lag: Realtime processing has stopped due to high system load. Some updates may be delayed.

Products

• VCF Operations for Networks

Issue/Introduction

1. Aria Operations for Networks shows Processing Grid Lag: Realtime processing has stopped due to high system load. Some updates may be delayed.
   
   Refer to error screenshots below:
   
   
   
   
   
2. System defined alert such as as below is seen on Aria Operations for Networks.
   
   
   Realtime processing has stopped due to high system load. Some updates may be delayed.
   Alert type: Problem
   Severity: Warning
   Manager: System Dashboard
   Defined by: System
   Alert tags: System Health
   System Health Message: Realtime processing has stopped due to high system load. Some updates may be delayed.
   Recommendation: It is a temporary condition. The system should recover and clear the error by itself. If this message persists for more than 12 hours, please contact VMware customer support.
3. Below queries and found out that found most of the routes deletions are happening from Fortinet Manager
   
   

   route group by vrf

   
   Screenshot below:

 

delete alerts group by Problem Entity.manager

Screenshot below:

 

 

Environment

Aria Operations for Networks 6.14.0

Cause

There are some VPN routes configured for some Fortinet devices which is causing high number of route changes and other Fortinet devices where VPN routes are not configured are also reporting high number of route changes. In a nutshell, Fortinet devices are reporting high number of route changes frequently.

Resolution

If you have encountered this issue, ensure you DO NOT perform any manual shutdown or reboot procedure of Platform Node(s). Capture details on given symptoms in Issue / Introduction section.

Open a support case with Broadcom Support to review your Aria Operations for Networks deployment. For more information, see Creating and managing Broadcom support cases.