"Query timed out after PT12S" - Aria Operations for Logs

Products

• VMware Aria Suite

Issue/Introduction

"Query timed out after PT12S" seen multiple times in the file: /storage/core/loginsight/var/runtime.log

[2024-05-06 08:32:40.876+0000] ["CassandraClientParallelExecutor-thread-1"/xx.xx.xx.xx WARN] [com.vmware.loginsight.cassandra.CassandraClient] [Exception during asynchronous execution (-2233411782988201016): Statement:(select guid, count 
from spock_cluster_counts where guid in ? and bucket = ? limit 1000000) java.util.concurrent.ExecutionException: com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT12S]
java.util.concurrent.ExecutionException: com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT12S

 

"No connection was available to Node" also seen multiple times in the file: /storage/core/loginsight/var/runtime.log

[2024-05-06 08:47:12.817+0000] ["CassandraClientParallelExecutor-thread-7241"/xx.xx.xx.xx WARN] [com.vmware.loginsight.cassandra.CassandraClient] [Exception during asynchronous execution (-122268630198556815): Statement:(select pattern, clusters 
from spock_patterns_v2 where bucket = ? limit 1000000) java.util.concurrent.ExecutionException: com.datastax.oss.driver.api.core.AllNodesFailedException: All 1 node(s) tried for the query failed (showing first 1 nodes, use getAllErrors() for more): Node(endPoint=xx.xx.xx.xx:9042, hostId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, hashCode=xxxxxxxx): [com.datastax.oss.driver.api.core.NodeUnavailableException: No connection was available to Node(endPoint=xx.xx.xx.xx:9042, hostId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, hashCode=xxxxxxxx)]]

 

There may also be intermittent issues when authenticating via API to "/api/v1/sessions" and "/api/v2/sessions", with the API responding with a 500 error.

Resolution

1. SSH to all nodes in the cluster as root.
2. Stop the loginsight service on all nodes in the cluster:
   
   systemctl stop loginsight
   
   
3. Backup the cassandra-driver.conf file on all nodes using the command below:
   
   cp -a /usr/lib/loginsight/application/lib/apache-cassandra-*/conf/cassandra-driver.conf /root
   
   
4. Run this command on all nodes:
   
   sed -i '/profiles {/i \ \ advanced.connection.pool.local.size = 4\n\ \ advanced.connection.pool.remote.size = 4\n' /usr/lib/loginsight/application/lib/apache-cassandra-*/conf/cassandra-driver.conf
   
   
5. Start the service again, primary node first, workers nodes second:
   
   systemctl start loginsight