AAI Jammer fails with "Caused by: com.termalabs.common.datastore.DataAccessException: Error while executing query: "from JobRun jr left join fetch jr.parent left join fetch jr.job job left join fetch job.type left join fetch jr.predecessors where ..."

Products

• Automation Analytics & Intelligence

Issue/Introduction

WARN [RunCycle] Jammer failed for scheduler: XXX: com.termalabs.server.cache.RunCacheNotInitializedException: Could not initialize run cache.

at com.termalabs.server.autosys.cache.AutosysNewRunCache.initializeData(AutosysNewRunCache.java:457)

at com.termalabs.server.autosys.cache.AutosysNewRunCache.initialize(AutosysNewRunCache.java:423)

at com.termalabs.server.autosys.cache.AutosysNewRunCache.getIncompleteJobRun(AutosysNewRunCache.java:471)

at com.termalabs.server.autosys.jammer.UncommittedRunCache.getIncompleteJobRun(UncommittedRunCache.java:169)

at com.termalabs.server.autosys.jammer.AutosysEventProcessorImpl.processRun(AutosysEventProcessorImpl.java:409)

at com.termalabs.server.autosys.jammer.AutosysEventProcessorImpl.processEvent(AutosysEventProcessorImpl.java:285)

at com.termalabs.server.publications.events.EventCollectingAutosysEventProcessor.processEvent(EventCollectingAutosysEventProcessor.java:30)

at com.termalabs.server.autosys.jammer.AutosysNewJammer.convertEventsToRuns(AutosysNewJammer.java:678)

at com.termalabs.server.autosys.jammer.AutosysNewJammer.processEvents(AutosysNewJammer.java:581)

at com.termalabs.server.autosys.jammer.AutosysNewJammer.processRuns(AutosysNewJammer.java:333)

at com.termalabs.server.system.RunCycle$JammerRunner.runJammer(RunCycle.java:1029)

at com.termalabs.server.system.RunCycle$JammerRunner.run(RunCycle.java:1011)

at com.termalabs.server.system.RunCycle.compute(RunCycle.java:281)

at com.termalabs.server.system.CommonJammerSequence.runCycle(CommonJammerSequence.java:225)

at com.termalabs.server.system.CommonJammerSequence.runJammerSequence(CommonJammerSequence.java:158)

at com.termalabs.server.system.JammerSequence.run(JammerSequence.java:136)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:750)

Caused by: com.termalabs.common.datastore.DataAccessException: Error while executing query: "from JobRun jr left join fetch jr.parent left join fetch jr.job job left join fetch job.type left join fetch jr.predecessors where   jr.scheduler.id = :schedulerId and jr.startTimeValue >= :fromTime and jr.futureRun = false order by jr.startTimeValue, jr.endTime"; schedulerId=UUID[8adbf1c6390baedc01390ce4324d000d], fromTime=2023/05/29 01:50:14.000 CDT

at com.termalabs.common.connection.hibernate.RichQueryCommon.executeAndTime(RichQueryCommon.java:1047)

at com.termalabs.common.connection.hibernate.RichQueryCommon.list(RichQueryCommon.java:969)

at com.termalabs.server.hibernate.RichQuery$CollectionQuery.asSetUnfiltered(RichQuery.java:175)

at com.termalabs.server.hibernate.RichQuery$CollectionQuery.asSet(RichQuery.java:155)

at com.termalabs.server.hibernate.jaws.HibernateJobRunData.getMostRecentRuns(HibernateJobRunData.java:574)

at com.termalabs.server.autosys.cache.AutosysNewRunCache$CacheData.initialize(AutosysNewRunCache.java:781)

at com.termalabs.server.autosys.cache.AutosysNewRunCache$CacheData.access$100(AutosysNewRunCache.java:722)

... 20 in common

Caused by: org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [com.termalabs.common.hibernate.jaws.JobRun#UUID[AYhv4xxxx_x=]]

at org.hibernate.impl.SessionFactoryImpl$2.handleEntityNotFound(SessionFactoryImpl.java:419)

at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:171)

at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:207)

at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)

at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:906)

at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:874)

at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:590)

at org.hibernate.type.EntityType.resolve(EntityType.java:412)

at org.hibernate.type.EntityType.nullSafeGet(EntityType.java:227)

at org.hibernate.persister.collection.AbstractCollectionPersister.readElement(AbstractCollectionPersister.java:725)

at org.hibernate.collection.PersistentSet.readFrom(PersistentSet.java:341)

at org.hibernate.loader.Loader.readCollectionElement(Loader.java:1031)

at org.hibernate.loader.Loader.readCollectionElements(Loader.java:669)

at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:614)

at org.hibernate.loader.Loader.doQuery(Loader.java:724)

at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)

at org.hibernate.loader.Loader.doList(Loader.java:2232)

at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2129)

at org.hibernate.loader.Loader.list(Loader.java:2124)

at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401)

at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:363)

at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)

at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1149)

at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)

at com.termalabs.common.connection.hibernate.RichQueryCommon$1.execute(RichQueryCommon.java:972)

at com.termalabs.common.connection.hibernate.RichQueryCommon$1.execute(RichQueryCommon.java:969)

... 27 in common

Environment

Release : 6.4.4

Resolution

Run the DB remediation script from the link below:

AAI Database Inconsistency Remediation Script

If you prefer to run the queries manually instead of running the script, the SQL queries that will likely resolve this issue are below:

 

References to missing predecessor runs

select count(*) from JobRun_PredJobRun where predJobRunId not in (select jobRunId from JobRun)

If any rows are returned, stop AAI and run the query SQL statement below:

delete from JobRun_PredJobRun

where predJobRunId not in (select jobRunId from JobRun);

 

On Oracle, make sure to commit the changes.