Aria Operations for Networks System Grid processing is running behind with Poor System Health, Poor System Infrastructure and Processing Lag of 4h

Products

• VMware Aria Operations for Networks

Issue/Introduction

Aria Operations for Networks installation shows problems.

• System Health: Poor
• Processing Lag: 4h+
• System Infrastructure: Poor

Operations for Networks System Grid processing is running behind (this is for far more than 12 hours the case - more like a couple of days)

GUI screenshots showing symptoms as below:

Alerts:

• Platform Health: Grid Processing Lag - Grid processing is running behind
• Platform Health: Real-time Processing is Suppressed - Real-time processing has stopped due to high system load. Some updates may be delayed.

 

High churn was seen in one object type. This churn should be investigated to determine if it is legitimate.

Screenshot for high churn:

Environment

Aria Operations for Networks 6.10.0
Aria Operations for Networks 6.11.0
Aria Operations for Networks 6.12.0
Aria Operations for Networks 6.12.0
Aria Operations for Networks 6.13.0

Cause

• There is a legitimate churn in one of the objects created in the AON database

• To mitigate churn, you can add/ allocate additional resources, e.g.

  • If system is deployed across a 3-node cluster, it is recommended to upgrade to a 5-node cluster

  • If NAV is not in use, it should be disabled to avoid causing system capacity breach errors. 

• Once capacity is resolved, there may be continued system alerts regarding a high load, which are gracefully handled. If you prefer to disable these alerts, this can be done by increasing the Protection Rate Limit from the default.

  • This change must be performed by Broadcom Support and may need to be re-applied post-upgrade

Resolution

1. Platform nodes increased 
2. Deployment type changed disable NAV 
3. Protection Rate Limit increased to limit alerts for high load

If needed, the Protection Rate Limit will need to be applied by  Broadcom Support and may need to be re-applied upon upgrade.

Contact Broadcom Support by opening support case to obtain assistance with Protection Rate reconfiguration

See Creating and managing Broadcom support cases.