Recurring Forced Traces with 'U00000002 EXISTS' Error

Products

CA Automic Workload Automation - Automation Engine CA Automic One Automation Automic SaaS

Issue/Introduction

The Automic Automation Engine system generates frequent forced memory traces and error messages during job activation or execution. This often occurs when large volumes of historical data persist in the database, causing Run-ID collisions.

Excerpt from the log:

U00011801 Error in Server routine 'JPAKTJ_R', Server: 'UC4#WP0XX' AE system: 'AESYSTEM'.
U00000002 EXISTS
U00015006 System forced memory trace dump.
U00015006 System forced memory trace dump.
U00003631 Dump caused by:
U00000002 EXISTS

The log may also include notes like:

U00011801 Error in Server routine 'JPEXEC_R', Server: 'UC4#WP0XX' AE system: 'AESYSTEM'.

Due to the frequency of these forced traces, It is not always possible to capture detailed traces (IP=2, DB=4) via Memory Trace.

Excerpt from the trace:

INSERT INTO AH (AH_Client, AH_OType, AH_SType, AH_Idnr, AH_USR_Idnr, AH_OH_Idnr, AH_TimeStamp1, AH_TimeStamp2, AH_MsgNr, AH_HostDst, AH_CodeNameDst, AH_FileNameDst, AH_CpuTime, AH_KernelTime, AH_UserTime, AH_IOCount, AH_HostSrc, AH_FileNameSrc, AH_Count, AH_FileSize, AH_Records, AH_Status, AH_RetCode, AH_ParentAct, AH_ParentPrc, AH_Restart, AH_RefNr, AH_CheckCount, AH_OccurCount, AH_Value, AH_DeleteFlag, AH_TaskCount, AH_Unit, AH_Category, AH_Archive2, AH_MaxRetCode, AH_ArchiveFlag, AH_Name, AH_Transferred, AH_ModCnt, AH_ProcFlag, AH_RunTime, AH_CmtFlag, AH_SkipMaxPar, AH_StartUC4, AH_RemoteStatus, AH_ERT, AH_ReuseHostGroup, AH_MaxParallelHG, AH_ChildCnt, AH_ChildCntFail, AH_ChildCntFailAll, AH_ChildCntRestart, AH_ErrorCount, AH_Resource, AH_TopNr, AH_EventId, AH_ModFlag, AH_CheckWithin, AH_InclSubDir, AH_ConsiderAll, AH_QueueSlot, AH_Sent, AH_Successful, AH_TotalFiles, AH_StreamSize, AH_Deployment, AH_WFType, AH_RollbackFlag, AH_RBCount, AH_RBErrCount, AH_SubDirStruct, AH_DeplDescIdnr, AH_AV_Mode, AH_ParentHir, AH_UC4Priority, AH_AJPP_Lnr) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
U00029108 UCUDB: SQL_ERROR Database handles DB-HENV: 1164c0490 DB-HDBC: 11653ae80
U00003591 UCUDB - DB error info: OPC: 'OCIStmtExecute' Return code: 'ERROR'
U00003592 UCUDB - Status: '' Native error: '1' Msg: 'ORA-00001: unique constraint (PK_AH) violated'

U00009909 TRACE: (Server-Inferface) 115085788 01296
00000000 25255352 30310004 00000002 4F434953 >%%SR01......OCIS<
00000010 746D7445 78656375 74657C45 52524F52 >tmtExecute|ERROR<
00000020 2020207C 7C4F5241 2D303030 30313A20 > ||ORA-00001: <
00000030 756E6971 75652063 6F6E7374 7261696E >unique constrain<
00000040 74202855 43345052 4431312E 504B4559 >t (PK_AH) violated <
00000060= 20202020 20202020 20202020 20202020 > <
00000400 20202020 20202020 20202020 00010000 > ....<

Environment

Product: Automic Automation Engine
Version: 12.3.x, 21.x, 24.x, 26.x
Database: Oracle, PostgreSQL, MS SQL
Platform: Distributed, Kubernetes

Cause

Unique constraint violations in the AH (Archive History) and EH (Execution History) tables occur due to Run-ID turnaround. This happens when the system attempts to assign a Run-ID that already exists in the tables because maintenance workflows (Reorg/Unload) have not removed older records.

To check if old records with superior runids than the current one exist on tables EH or AH (and subsequently also in the related tables), you can use the following queries adapting to your environment/database:

For table EH:
select EH_AH_idnr, EH_StartTime, EH_status from EH where EH_AH_idnr > (select ids_act from IDS where IDS_Name = 'AH')
For table AH:
select ah_client,ah_idnr,ah_timestamp1,ah_status from ah where ah_idnr > (select ids_act from ids where ids_name ='AH') order by ah_idnr ASC

Resolution

It's important to maintain the database records in the AH table. The following may be causes for the AH records staying when they should be removed:

Old data in the EH table (Process Monitoring) causes the AH entries to not be removed. To resolve this:
- Access the Process Monitoring perspective in the AWI while logged into Client 0. From this perspective, you can view tasks from all clients
- Filter tasks with statuses of ANY_OK and ANY_ABEND.
- Click the column header to sort the filtered tasks by either Start Time or End Time. In order to find older records.
- Look for unusually old entries in the task list. Check the date and time stamps carefully. These older records are likely causing the problem.
- Select the old tasks you identified and deactivate them directly in Process Monitoring. Right-click the selected tasks --> Modify Task --> choose "Deactivate."
Database sequence turnover has occurred. This should be resolved by running the database maintenance.
The Database Maintenance does not remove the database records as expected. Look at the maintenance logs (archive, reorg, unload) to see if they are removing records as expected. Look at the Troubleshooting guide - specific record not being removed from ah/rh table and should be knowledge article for help in understanding this

Important: Consistent housekeeping of Process Monitoring data, coupled with regular database reorganization (Re-org), mitigates the risk of encountering these violations.

In case you find millions of records in the EH or AH tables that need to be deleted in bulk and need assistance for the best strategy to delete them, please reach out to Technical Support with this KB.

Additional Information

DEACTIVATE_UC_OBJECT

Stopping and Interrupting