Consistent Prediction Timeout messages when AAI database is in a different data center then AAI

Products

• Automic Automation Intelligence

Issue/Introduction

jaws.log shows consistent timeout messages like below consistently:

WARN [ForecastJobStreamTask] Prediction for <Jobstream name> at 2020/12/03 23:30:00.000 EST timed out. Timeout period is 30

WARN [NewPooledConnection] [c3p0] A PooledConnection that has already signalled a Connection error is still in use!

WARN [NewPooledConnection] [c3p0] Another error has occurred [ java.sql.SQLRecoverableException: Closed Connection ] which will not be reported to listeners!: java.sql.SQLRecoverableException: Closed Connection

WARN [AbstractBatcher] exception clearing maxRows/queryTimeout: java.sql.SQLRecoverableException: Closed Connection

ERROR [PredictionEngineWorker] Error in prediction: com.termalabs.common.datastore.DataAccessException: Error while executing query: "from JobStreamRun jsr left join fetch jsr.jobStreamRunJobRuns where jsr.jobStream.id = :jobStreamId and jsr.initialJobRunStartTime = (select max(jsr.initialJobRunStartTime) from JobStreamRun jsr where jsr.jobStream.id = :jobStreamId and jsr.complete = true)"; jobStreamId=UUID[xxxxx]

Environment

Release : 6.0.2

Component : AUTOMIC AUTOMATION INTELLIGENCE ENGINE

Resolution

Certain predictions will do database queries for incomplete job stream runs because they are not in the cache.

 

When these queries are slow, the predictions will be as well, and likely are the cause of the prediction timeout

 

Normally all necessary runs are in the cache, but that assumes steady-state operation with up-to-date / current event data.

When the database is in a data center in another region, it can cause latency which could result in timeouts when calculating predictions. 

The database should always be in the same data center as the AAI server.