AAI Database Inconsistency Queries for when AAI doesn't fully start up
search cancel

AAI Database Inconsistency Queries for when AAI doesn't fully start up

book

Article ID: 205932

calendar_today

Updated On:

Products

Automic Automation Intelligence

Issue/Introduction

AAI not fully initializing after startup, and is stuck running the same queries over and over again.

There can be several scenarios that cause this.  

 

Below are some examples:

 

During startup log shows 

WARN [CurrentStateVendorImpl] Invalid data found in CurrentState table.  Will attempt fixup.

INFO  [QueryTimingService] <80> "from CurrentStateThin where mostRecentRunId is not null"

 

QueryTimingService messages repeatedly show:

 INFO [QueryTimingService] 100.0% of last 0:01:00 spent in DB/API operations (possibly on multiple threads/overlapping queries)
  100.0%: 8x (avg. time: 7.501) query: select id from JobRun where id in (:ids)

 

 

Log messages like:

WARN  [JawsDbIntegrityUtil] Found 904 jobRunIds in the RecentStateType table that did NOT exist. Attempting to correct.

 

You may also see:

ERROR [JammerManager] Interrupted: com.termalabs.server.cache.RunCacheNotInitializedException: Could not initialize run cache.
 at com.termalabs.server.autosys.cache.AutosysNewRunCache.initializeData(AutosysNewRunCache.java:457)
 at com.termalabs.server.autosys.cache.AutosysNewRunCache.initialize(AutosysNewRunCache.java:423)
 at com.termalabs.server.autosys.cache.AutosysNewRunCache.getMostRecentlyStartedRunForEachJob(AutosysNewRunCache.java:388)
 at com.termalabs.server.cache.JobRunVendorImpl.loadMostRecentRuns(JobRunVendorImpl.java:789)
 at com.termalabs.server.cache.JobRunVendorImpl.setUpMostRecentRuns(JobRunVendorImpl.java:703)
 at com.termalabs.server.system.JammerManager.doFirstTimeProcessing(JammerManager.java:403)
 at com.termalabs.server.system.JammerManager.run2(JammerManager.java:308)
 at com.termalabs.server.system.JammerManager.run(JammerManager.java:238)
 at com.termalabs.server.system.ApplicationControllerService$2.run(ApplicationControllerService.java:268)
 at java.util.TimerThread.mainLoop(Timer.java:555)
 at java.util.TimerThread.run(Timer.java:505)
Caused by: com.termalabs.common.datastore.DataAccessException: Error while executing query: "from JobRun jr left join fetch jr.parent left join fetch jr.job job left join fetch job.type left join fetch jr.predecessors where   jr.scheduler.id = :schedulerId and jr.startTimeValue >= :fromTime and jr.futureRun = false order by jr.startTimeValue, jr.endTime"; schedulerId=UUID[AXoftDMHAHU=], fromTime=2022/05/18 10:59:41.000 PDT
 at com.termalabs.common.connection.hibernate.RichQueryCommon.executeAndTime(RichQueryCommon.java:1047)
 at com.termalabs.common.connection.hibernate.RichQueryCommon.list(RichQueryCommon.java:969)
 at com.termalabs.server.hibernate.RichQuery$CollectionQuery.asSetUnfiltered(RichQuery.java:175)
 at com.termalabs.server.hibernate.RichQuery$CollectionQuery.asSet(RichQuery.java:155)
 at com.termalabs.server.hibernate.jaws.HibernateJobRunData.getMostRecentRuns(HibernateJobRunData.java:572)
 at com.termalabs.server.autosys.cache.AutosysNewRunCache$CacheData.initialize(AutosysNewRunCache.java:781)
 at com.termalabs.server.autosys.cache.AutosysNewRunCache$CacheData.access$100(AutosysNewRunCache.java:722)
 ... 10 in common
Caused by: org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [com.termalabs.common.hibernate.jaws.JobRun#UUID[AYDdRABXDOA=]]

 

Environment

Release : 6.0.2

Component : AUTOMIC AUTOMATION INTELLIGENCE ENGINE

Resolution

There are some known data inconsistency scenarios sometimes occur.

 

If you are running 6.4.2-2 or later you can follow the steps below to run these steps automatically.

 

1. Stop the AAI service

 

2. Go to the <Install Dir>/config director on your AAI server.

 

3. Run the following command:

java -Dswing.aatext=true -jar config.jar datainconsistency --remediate <Path to <Install Dir>/jboss directory>

 

For example if AAI is installed at /opt/aai your command would be:

java -Dswing.aatext=true -jar config.jar datainconsistency --remediate /opt/aai/jboss

 

You should see output like below, if anything is updated it will display in the output. 

Starting Config Tool
Checking if the Automic Automation Intelligence (AAI) server is running
The Automic Automation Intelligence (AAI) server is not running
Fixing Data Inconsistencies. Please do no start AAI service while running fix
Fixing missing parent JobRuns
Fixed missing parent JobRuns. 0 rows affected.
Fixing missing predecessor runs
Fixed missing predecessor runs. 0 rows affected.
Fixing missing successor JobRuns
Fixed missing successor JobRuns. 0 rows affected.
Fixing missing job runs in JobStreamRuns
Fixed missing job runs in JobStreamRuns. 0 rows affected.
Fixing missing JobRun references in RecentState
Fixing missing JobRun references in RecentState. 0 rows affected.
Fixing invalid dependent job condition reference
Fixed invalid dependent job condition reference. 0 rows affected.
All Data Inconsistencies fixed. Exiting..

 

4. Start up the AAI service.

 

If you are on an earlier release follow the steps below:

If you should run into this scenario run the select queries below. If any queries return a non zero result, run the update or delete statements listed below them.

Only start up services after all the select queries return 0.



1. References to missing parent job runs

select s.schedulerName, count(*) from JobRun jr, JobScheduler s where jr.parentJobRunId not in (select jobRunId from JobRun) and jr.jobSchedulerId = s.jobSchedulerId group by s.schedulerName


-- resolution

update JobRun

set parentJobRunId = null

where parentJobRunId not in (select jobRunId from JobRun)



2. References to missing predecessor runs

select count(*) from JobRun_PredJobRun where predJobRunId not in (select jobRunId from JobRun)

--resolution

delete from JobRun_PredJobRun

where predJobRunId not in (select jobRunId from JobRun)




3. Missing successor job runs

select count(*) from JobRun_PredJobRun where jobRunId not in (select jobRunId from JobRun)


-resolution

delete from JobRun_PredJobRun

where jobRunId not in (select jobRunId from JobRun)



4. Missing job runs in jobstream runs

select count(*) from JobStreamRun_JobRun where jobRunId not in (select jobRunId from JobRun)


--resolution

delete from JobStreamRun_JobRun

where jobRunId not in (select jobRunId from JobRun)




5. Missing job run references in recent state

select count(*) from RecentStateType where jobRunId not in (select jobRunId from JobRun)

 

--resolution

update RecentStateType

set jobRunId = null

where jobRunId not in (select jobRunId from JobRun)

 


6. Invalid dependent job condition reference

select count(*) from JobCondition where dependantJobId not in (select jobId from JawsJob)


--resolution

select s.schedulerName, jj.jobId, jj.jobName from JawsJob jj, JobScheduler s where jj.jobName LIKE 'missing%' and jj.jobSchedulerId = s.jobSchedulerId




update JobCondition

set dependantJobId = 'MISSING_JOB_ID for appropriate SCHEDULER'

where dependantJobId not in (select jobId from JawsJob)

Additional Information

See also the KB article below for another database issue that could prevent AAI from starting:

https://knowledge.broadcom.com/external/article?articleId=194479

Powered by Wolken Software