ORA-20056: Could not lock jobid 14364954 in 30 seconds in 30 tries status=28-PRED WAIT

Products

• CA Automic Applications Manager (AM)

Issue/Introduction

After upgrading and starting Applications Manager the following error may be seen in the RmiServer logs:

ErrorMsg: AwE-5001 Database Query Error 
Details: ORA-20056: Could not lock jobid 14364954 in 30 seconds in 30 tries status=31-INITIATED
ORA-06512: at "UC4USR.AW5", line 805
ORA-06512: at "UC4USR.AW5", line 992
ORA-06512: at line 1

aw5.aw_set_job_status
 0 jobid: IN:NUMERIC:java.math.BigDecimal:14364954
 1 status: IN:OUT:INTEGER:java.lang.Integer:32
 2 currentStatus: IN:INTEGER:java.lang.Integer:31

  sID-895485572 
java.sql.SQLException: ORA-20056: Could not lock jobid 14364954 in 30 seconds in 30 tries status=31-INITIATED
ORA-06512: at "UC4USR.AW5", line 805
ORA-06512: at "UC4USR.AW5", line 992
ORA-06512: at line 1

aw5.aw_set_job_status
 0 jobid: IN:NUMERIC:java.math.BigDecimal:14364954
 1 status: IN:OUT:INTEGER:java.lang.Integer:32
 2 currentStatus: IN:INTEGER:java.lang.Integer:31

 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:494)
 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:446)
 at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1054)
 at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:623)
 at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
 at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
 at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:223)
 at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:56)
 at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:907)
 at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1119)
 at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
 at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1300)
 at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3887)
 at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4230)
 at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1079)
 at com.appworx.server.sql.A.B(DataAPI.java:79)
 at com.appworx.server.sql.DBAccess.A(DBAccess.java:984)
 at com.appworx.server.sql.DBAccess.getProcedureData(DBAccess.java:1041)
 at com.appworx.server.sql.DBAccess.getProcedureData(DBAccess.java:1056)
 at com.appworx.server.sql.CommonDB.getProcedureData(CommonDB.java:377)
 at com.appworx.server.sql.CommonDB.getProcedureData(CommonDB.java:366)
 at com.appworx.server.data.MasterDB.setJobStatus(MasterDB.java:4759)
 at com.appworx.server.data.JobsData.setJobStatusNow(JobsData.java:1473)
 at com.appworx.master.JobStarter.s(JobStarter.java:2420)
 at com.appworx.master.JobStarter.init(JobStarter.java:402)
 at com.appworx.server.data.JobsData.ΨJobsData.java:1068)
 at com.appworx.server.data.JobsData.A(JobsData.java:76)
 at com.appworx.server.data.JobsData$_E.A(JobsData.java:3430)
 at com.appworx.server.data.JobsData$_E.doRun(JobsData.java:3474)
 at com.uc4.be.threading.AbstractWorker.run(AbstractWorker.java:367)
 at java.lang.Thread.run(Unknown Source)

Environment

Release : 9.3

Component : APPLICATIONS MANAGER

Cause

This issue usually occurs with locked objects in oracle database.

Resolution

1. Stop all Applications Manager processes
2. Refer to DBA to check Applications Manager schema for any locked objects and unlock if possible
3. Delete erroneous JobID from the Backlog using sql. The JobID(s) can be seen in the error. Example sql based on above error:
   
   SQL> delete from so_job_queue where so_jobid='14364954';
   SQL> commit;
   
   
4. Start Applications Manager processes

Additional Information

It is recommended to take a backup of the AM database before running any manual sql updates.