Jobs not running due to out of order parent jobids - Applications Manager

Products

• Automic Applications Manager

Issue/Introduction

Applications Manager may stop processing jobs following a system crash, network outage, or running out of Java memory.

Symptoms
In the RmiServer logs, you will see recurring errors similar to: Jobid #### came in out of order, parent jobid is ####

This occurs when a job in the backlog or history cannot find its parent jobid, causing the RmiServer to enter a processing loop. This often results in high CPU usage and subsequent Java heap errors.

Environment

Applications Manager

Cause

The "out of order" error is typically a secondary symptom of an earlier failure (like a Java OutOfMemory error) that prevented the parent job's state from being correctly recorded.

Resolution

1. Prerequisite: If the issue was caused by a Java memory failure, first follow the steps in KB 88261: Increasing Java Memory available to Applications Manager to increase the memory allocation.
2. Stop the Applications Manager processes.
3. Log into SQL-Plus and run the following queries to identify the problematic records:
   
   sql

   select count(*) from so_job_queue where so_jobid='<Jobid_from_error>';
   select count(*) from so_job_history where so_jobid='<Parent_jobid_from_error>';

4. Remove the offending records:
   sql

   delete from so_job_queue where so_jobid='<Jobid_from_error>';
   delete from so_job_history where so_jobid='<Parent_jobid_from_error>';
   commit;

5. Restart the Applications Manager processes.

Additional Information

For instructions on how to download the latest product releases or patches, see Download Broadcom Products and Software.

To speak with a customer representative or a Support Engineer, see Contact Support. Scroll to the bottom of the page and click on your respective region.