AE system freezes with ORA-00600: internal error code, arguments: [ktsbnew:objdchk_kcbnew_3

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

Ocassionally, an Automation Engine with an Oracle Database gets into No Responsive state, with the following error appearing in the WP logs:

20230211/235241.528 - U00029108 UCUDB: SQL_ERROR    Database handles  DB-HENV: 40aa300  DB-HDBC: 5472cd0
20230211/235241.528 - U00003591 UCUDB - DB error info: OPC: 'OCIStmtExecute' Return code: 'ERROR'
20230211/235241.528 - U00003592 UCUDB - Status: '' Native error: '600' Msg: 'ORA-00600: internal error code, arguments: [ktsbnew:objdchk_kcbnew_3], [], [], [], [], [], [], [], [], [], [], []'
20230211/235241.528 - U00003536 UCUDB: FATAL DATA BASE ERROR: Re-connection will be attempted in 10 seconds.
20230211/235241.796 - U00003538 UCUDB: Re-connection to database successful. Processing will continue.
20230211/235245.797 - U00000006 DEADLOCK or Connection to database lost - Rollback handling initiated. See previous messages.
20230211/235245.797 - U00003594 UCUDB Ret: '6' opcode: 'INPK' SQL Stmnt: 'INSERT INTO MQ1OWP (MQOWP_PK, MQOWP_System, MQOWP_CAddr, MQOWP_CSRName, MQOWP_CAcv, MQOWP_BAddr, MQOWP_BSRName, MQOWP_BAcv, MQOWP_FAddr, MQOWP_LogAddr, MQOWP_PhysAddr, MQOWP_BTable, MQOWP_SchedTime, MQOWP_Status, MQOWP_Priority, MQOWP_DRole, MQOWP_LAddr, MQOWP_Len, MQOWP_Msg) VALUES (SQ_MQ1OWP.nextval, ?, ?, ?, ?, NULL, NULL, ?, ?, NULL, NULL, ?, sys_extract_utc(systimestamp(0)), ?, ?, ?, NULL, ?, ?) RETURNING MQOWP_PK INTO ?'
20230211/235245.797 - U00003524 UCUDB: ===> Time critical DB call!       OPC: 'INPK' time: '4:952.899.999'
20230211/235245.797 - U00003525 UCUDB: ===> 'INSERT INTO MQ1OWP (MQOWP_PK, MQOWP_System, MQOWP_CAddr, MQOWP_CSRName, MQOWP_CAcv, MQOWP_BAddr, MQOWP_BSRName, MQOWP_BAcv, MQOWP_FAddr, MQOWP_LogAddr, MQOWP_PhysAddr, MQOWP_BTable, MQOWP_SchedTime, MQOWP_Status, MQOWP_Priority, MQOWP_DRole, MQOWP_LAddr, MQOWP_Len, MQOWP_Msg) VALUES (SQ_MQ1OWP.nextval, ?, ?, ?, ?, NULL, NULL, ?, ?, NULL, NULL, ?, sys_extract_utc(systimestamp(0)), ?, ?, ?, NULL, ?, ?) RETURNING MQOWP_PK INTO ?'
20230211/235245.797 - U00000006 DEADLOCK or Connection to database lost - Rollback handling initiated. See previous messages

In most cases the problem its auto-resolved, but in other occasions causes the system to freeze and in such case a Manual stop and Cold Start are required to restore the service.

Environment

Release : 12.x and 21.x

Component: Automation Engine:

Database: Oracle

Cause

Oracle database bug, according to Oracle Support the call stack is a close match to Bug 29967980 - ORA-00600 [ktspfmdb:objdchk_kcbnew_3] on DML Query ( Doc ID 29967980.8), 

Resolution

Solution:

Apply the associated fix for Bug 29967980 - ORA-00600 [ktspfmdb:objdchk_kcbnew_3]

 

Workaround:

Cold start of the automation engine to perform a truncate of all the mq tables should avoid the problem for some time, but it will occur again until the Oracle Database is patched.