AWA: Recurring Forced Traces with 'U00000002 EXISTS' Error
search cancel

AWA: Recurring Forced Traces with 'U00000002 EXISTS' Error

book

Article ID: 400356

calendar_today

Updated On:

Products

CA Automic Workload Automation - Automation Engine CA Automic One Automation Automic SaaS

Issue/Introduction

It is noticed that the system is constantly generating forced traces

Excerpt from the log:

U00011801 Error in Server routine 'JPAKTJ_R', Server: 'UC4#WP0XX' AE system: 'AESYSTEM'.
U00000002 EXISTS
U00015006 System forced memory trace dump.
U00015006 System forced memory trace dump.
U00003631 Dump caused by: 
U00000002 EXISTS

Due to the frequency of these forced traces, It is not always possible to capture detailed traces (IP=2, DB=4) via Memory Trace.

Excerpt from the trace:

INSERT INTO AH (AH_Client, AH_OType, AH_SType, AH_Idnr, AH_USR_Idnr, AH_OH_Idnr, AH_TimeStamp1, AH_TimeStamp2, AH_MsgNr, AH_HostDst, AH_CodeNameDst, AH_FileNameDst, AH_CpuTime, AH_KernelTime, AH_UserTime, AH_IOCount, AH_HostSrc, AH_FileNameSrc, AH_Count, AH_FileSize, AH_Records, AH_Status, AH_RetCode, AH_ParentAct, AH_ParentPrc, AH_Restart, AH_RefNr, AH_CheckCount, AH_OccurCount, AH_Value, AH_DeleteFlag, AH_TaskCount, AH_Unit, AH_Category, AH_Archive2, AH_MaxRetCode, AH_ArchiveFlag, AH_Name, AH_Transferred, AH_ModCnt, AH_ProcFlag, AH_RunTime, AH_CmtFlag, AH_SkipMaxPar, AH_StartUC4, AH_RemoteStatus, AH_ERT, AH_ReuseHostGroup, AH_MaxParallelHG, AH_ChildCnt, AH_ChildCntFail, AH_ChildCntFailAll, AH_ChildCntRestart, AH_ErrorCount, AH_Resource, AH_TopNr, AH_EventId, AH_ModFlag, AH_CheckWithin, AH_InclSubDir, AH_ConsiderAll, AH_QueueSlot, AH_Sent, AH_Successful, AH_TotalFiles, AH_StreamSize, AH_Deployment, AH_WFType, AH_RollbackFlag, AH_RBCount, AH_RBErrCount, AH_SubDirStruct, AH_DeplDescIdnr, AH_AV_Mode, AH_ParentHir, AH_UC4Priority, AH_AJPP_Lnr) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
U00029108 UCUDB: SQL_ERROR    Database handles  DB-HENV: 1164c0490  DB-HDBC: 11653ae80
U00003591 UCUDB - DB error info: OPC: 'OCIStmtExecute' Return code: 'ERROR'
U00003592 UCUDB - Status: '' Native error: '1' Msg: 'ORA-00001: unique constraint (AESYSTEM.PKEY_AH) violated'

U00009909 TRACE: (Server-Inferface)            115085788 01296
          00000000  25255352 30310004 00000002 4F434953  >%%SR01......OCIS<
          00000010  746D7445 78656375 74657C45 52524F52  >tmtExecute|ERROR<
          00000020  2020207C 7C4F5241 2D303030 30313A20  >   ||ORA-00001: <
          00000030  756E6971 75652063 6F6E7374 7261696E  >unique constrain<
          00000040  74202855 43345052 4431312E 504B4559  >t (AESYSTEM.PKEY<
          00000050  5F454829 2076696F 6C617465 64202020  >_EH) violated   <
          00000060= 20202020 20202020 20202020 20202020  >                <
          00000400  20202020 20202020 20202020 00010000  >            ....<

Environment

Release: 12.3.x 21.x 24.x

Component: Automic Automation 

Cause

Forced traces indicate a unique constraint violation occurring on the AH and EH tables. This violation typically arises from run-ID turnaround in the customer environment, where older records with previously used run-IDs persist in these tables.

To check if old records with superior runids than the current one exist on tables EH or AH (and subsequently also in the related tables), you can use the following queries adapting to your environment/database:

  • For table EH:
    select EH_AH_idnr, EH_StartTime, EH_status from EH where EH_AH_idnr > (select ids_act from IDS where IDS_Name = 'AH') 

  • For table AH:
    select ah_client,ah_idnr,ah_timestamp1,ah_status from ah where ah_idnr > (select ids_act from ids where ids_name ='AH') order by ah_idnr ASC

Resolution

  • Access the Process Monitoring perspective in the AWI while logged into Client 0. From this perspective, you can view tasks from all clients
  • Filter tasks with statuses of ANY_OK and ANY_ABEND.
  • Click the column header to sort the filtered tasks by either Start Time or End Time. In order to find older records.
  • Look for unusually old entries in the task list. Check the date and time stamps carefully. These older records are likely causing the problem.
  • Select the old tasks you identified and deactivate them directly in Process Monitoring. Right-click the selected tasks --> Modify Task --> choose "Deactivate."
  • Run the Re-org maintenance workflows to remove the deactivated records from the database. 

Important: Consistent housekeeping of Process Monitoring data, coupled with regular table reorganization (Re-org), mitigates the risk of encountering these violations.

In case you find millions of records in the EH or AH tables that need to be deleted in bulk and need assistance for the best strategy to delete them, please reach out to Technical Support with this KB.

Additional Information

Powered by Wolken Software