High CPU usage / CPU spike (more than 30-40 GHz) is seen on Aria Operations for Logs cluster

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• CPU usage at 100% is observed on 3 or more Aria Operations for Logs nodes Cluster and due to this other VMs in the vCenter cluster are being affected in terms of performance.
• The environment may have 2-3 vCenter servers integrated with around 65 hosts.
• Even though there are 5 Nodes, each worker node may still be consuming nearly 25 GHz or more which may affect other VMs in the vCenter cluster. Find the relevant screenshot below:

 

Environment

• Aria Operations for Logs 8.x

Cause

• The CPU Spike occurs when Precomputation Queries are generated too often.
• Precomputation Queries are an automated process in Aria Operations for Logs which regularly runs user-defined queries to prime the cache so whenever users run queries, the request is fulfilled by the cache.
• To mitigate this, cache priming needs to be disabled. An example of high Precomputation queries is below:

Resolution

To address this issue disable cache priming on the cluster by following:

1. Open https://<OperationsForLogs>/internal/config, and check the "Show all settings" checkbox.
   
   
2. Go to the end of the XML, and add the following sub-section into the <config></config> section:
   
   <searcher>       
     <num-queries-to-prime-cache value="0" />     
   </searcher>
   
   
3. Save the configuration and close the page.
   
   
4. Open the same page and make sure your configuration still exists (it may have been moved into a different location, so you'll have to search for it via browser 'find' tool).
   
   
5. Restart services on every cluster node one by one:
   service loginsight stop
   service loginsight start

Additional Information

High CPU on Aria Operations for Logs nodes can also be caused by corruption in the Cassandra database:

• FAILED: Unable to get user data. Possible Cassandra is down