Performance issues in Aria Operations for Logs even after scale up

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Experiencing slowness in Aria Operations for Logs UI 
• Cluster is still undersized even after adding additional nodes as number events increase due to dropped events.  Recommendation for sizing the appliance can be found in the following document Sizing the VMware Aria Operations for Logs Virtual Appliance 
• See a large amount of incoming and dropped API events when navigating to Management > System monitor > Statistics > Show advanced statistics    

Environment

Aria Operations for Logs 8.18.x

Cause

Forwarding rule is configured to forward logs from Aria Operations for Logs instance A to Aria Operations for Logs Instance B and Aria Operations for Logs B is forwarding logs back to Aria Operations for Logs A causing a forwarding loop which results in the cluster being undersized as a result experience performance issues in both clusters. 

Resolution

Locate the offending forwarding rules and adjust the rule to prevent all logs from forwarding to the Aria Operations for logs instances, advise would be to consider using tagging to reduce the amount of logs forwarding also.  The following documentation would be useful for making the adjustment Add a VMware Aria Operations for Logs Log Forwarding Destination.