Possible cause of Jobs not running

Products

• CA Automic Applications Manager (AM)

Issue/Introduction

Possible cause of Jobs not running

Environment

Applications Manager 9.4.x and 9.5.x

Resolution

Error Details

Jobid 222939 came in out of order, parent jobid is 222937
Jobid 222940 came in out of order, parent jobid is 222937
Jobid 222941 came in out of order, parent jobid is 222937
Jobid 222942 came in out of order, parent jobid is 222937

---

Symptoms

Applications Manager may stop running jobs and you see the above errors in the logs after a system crash, network outage or running out of Java memory.  This error can be seen if there is a job in the backlog or history which cannot find its parent jobid, which causes the RmiServer to go into a loop and can possibly cause java memory to fill up.

 

---

Solution

The jobs may need to be removed from the so_job_queue_activity, so_job_queue and/or the so_job_history tables. 

Please log into SQL-Plus and run the following select statements.  Replace <jobid> for the actual jobid referenced in the error message and send Automic support the results:

select count(distinct so_status_name), so_status_name from so_job_history
where so_status_name in('INITIATED','RUNNING','STAGED','STAGED_PW','STARTED','STARTING','QUEUE WAIT','PRED WT HOLD','PRED WAIT','LAUNCH ERROR','KILLING','DATE PENDING','CONDITN WAIT','AGENT WAIT')group by so_status_name, so_status order by 2;

select count(*) from so_job_queue where so_jobid=<jobid>;
select count(*) from so_job_history where so_jobid=<jobid>;
select count(*) from aw_job_queue_activity where so_jobid=<jobid>;