AAI not fully starting: ERROR [JammerManager] Interrupted: com.termalabs.server.cache.UncorrectedCurrentStateTableException: Unable to read CurrentState table after correcting data.

Products

• Automic Automation Intelligence

Issue/Introduction

AAI will not fully initialize and the error below is seen in the jaws.log

 

ERROR [JammerManager] Interrupted: com.termalabs.server.cache.UncorrectedCurrentStateTableException: Unable to read CurrentState table after correcting data.
 at com.termalabs.server.cache.CurrentStateVendorImpl.repairCurrentStates(CurrentStateVendorImpl.java:342)
 at com.termalabs.server.cache.CurrentStateVendorImpl.initializeAllCurrentStates(CurrentStateVendorImpl.java:312)
 at com.termalabs.server.system.JammerManager.doFirstTimeProcessing(JammerManager.java:402)
 at com.termalabs.server.system.JammerManager.run2(JammerManager.java:309)
 at com.termalabs.server.system.JammerManager.run(JammerManager.java:239)
 at com.termalabs.server.system.ApplicationControllerService$2.run(ApplicationControllerService.java:268)
 at java.util.TimerThread.mainLoop(Timer.java:555)
 at java.util.TimerThread.run(Timer.java:505)
Caused by: com.termalabs.common.datastore.DataAccessException: Error while executing query: "from CurrentState cs left join fetch cs.mostRecentRun left join fetch cs.mostRecentRun.parent"
 at com.termalabs.common.connection.hibernate.RichQueryCommon.executeAndTime(RichQueryCommon.java:1047)
 at com.termalabs.common.connection.hibernate.RichQueryCommon.list(RichQueryCommon.java:969)
 at com.termalabs.server.hibernate.RichQuery$CollectionQuery.asSetUnfiltered(RichQuery.java:175)
 at com.termalabs.server.hibernate.RichQuery$CollectionQuery.asSet(RichQuery.java:155)
 at com.termalabs.server.hibernate.jaws.HibernateCurrentStateData.getAllCurrentStates(HibernateCurrentStateData.java:81)
 at com.termalabs.server.cache.CurrentStateVendorImpl.getAllCurrentStates(CurrentStateVendorImpl.java:350)
 ... 7 in common
Caused by: org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [com.termalabs.common.hibernate.jaws.JobRun#UUID[AXpBWV_yD0M=]]
 at org.hibernate.impl.SessionFactoryImpl$2.handleEntityNotFound(SessionFactoryImpl.java:419)
 at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:171)
 at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:207)
 at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
 at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:906)
 at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:874)
 at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:590)
 at org.hibernate.type.EntityType.resolve(EntityType.java:412)
 at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:139)
 at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:877)
 at org.hibernate.loader.Loader.doQuery(Loader.java:752)
 at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
 at org.hibernate.loader.Loader.doList(Loader.java:2232)
 at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2129)
 at org.hibernate.loader.Loader.list(Loader.java:2124)
 at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401)
 at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:363)
 at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
 at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1149)
 at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
 at com.termalabs.common.connection.hibernate.RichQueryCommon$1.execute(RichQueryCommon.java:972)
 at com.termalabs.common.connection.hibernate.RichQueryCommon$1.execute(RichQueryCommon.java:969)
 ... 13 in common

Environment

AAI all versions

Cause

Database inconsistencies detected on start up.

Resolution

1. Run the select statement below to identify if there are database inconsistencies for JobRun's.

 

select s.schedulerName, count(*) from JobRun jr, JobScheduler s where jr.parentJobRunId not in (select jobRunId from JobRun) and jr.jobSchedulerId = s.jobSchedulerId group by s.schedulerName

 

2. If you get a result, stop the AAI service and verify jboss is no longer running "ps -eaf | grep jboss".

 

3. Run the following SQL statement:

update JobRun

set parentJobRunId = null

where parentJobRunId not in (select jobRunId from JobRun);

commit;

 

4. When the update completes attempt to start the AAI service and verify the error is no longer present in the jaws.log.

Additional Information

See also the kb below if you run into other errors during startup:

https://knowledge.broadcom.com/external/article?articleId=205932