Jobs in various statuses remain stuck in Backlog

Products

• Automic Applications Manager

Issue/Introduction

Job(s) display in various statuses such as RUNNNING, STAGED, INITIATED, etc, in the backlog even though the related scripts have completed at the system level.

Environment

Release: CA Automic Applications Manager (AM) version 8.X.X and version 9.XX.

Cause

The RmiServer Process is no longer evaluating the Jobs and they remain hung in a RUNNING status.

Resolution

Troubleshooting Steps

-Enable the RmiServer debugging. From the Java UI Client-> Help-> About Application Manager ->Click on Debug-> Check the server

-Leave it in debugging mode for about 5 minutes
 
-Disable the debugging from the Java UI
 
 
Then,please send us the following:
 
-The complete and zipped log directory of the RmiServer
 
-The result of the command below:
 
jstack <Process ID of the com.appworx.server.data.AxRmiServer >
 
-The screenshot of the list of the jobs that are stuck in RUNNING status
 
-The agentservice<timestamp> of the agent where the jobs executed

 

Workaround

Perform the following either of the following steps, ranked from least intrusive to most intrusive, in order to get the RUNNING status Jobs, as well as other stuck status such as the KILLING status, out of the queue:

1. Stopping/Restarting the Automic Application Engineer (Master):
   • Select the Automic Application Engine (Master) from the Agent Summary Window, right click and select Stop and and then repeat to Start once in Stopped status.
     • ***NOTE: This may cause the Master to re-evaluate the status of the Jobs and move them to the history***
2. Stopping/Restating the RmiServer Process:
   • From the command line with “sosite” invoked issue the following commands:
     1. stopso rmi
     2. startso rmi
   • ***NOTE: This will require all User’s to restart their instance of the Automic Application Manager (AM) Client***
3. Delete the Jobs from the backlog utilizing a SQL statement:
   • As this option can lead to potential adverse effects on the Automic Application Manager (AM) instance, please contact Automic Support and reference this Knowledge Base (KB) Article and number to obtain the necessary SQL statements.