Autosys job with "EVENT_HLDR_ERROR", Job didn't run as schedule.
search cancel

Autosys job with "EVENT_HLDR_ERROR", Job didn't run as schedule.

book

Article ID: 199774

calendar_today

Updated On:

Products

CA Workload Automation AE - Business Agents (AutoSys) CA Workload Automation AE - System Agent (AutoSys) CA Workload Automation AE - Scheduler (AutoSys) Workload Automation Agent Autosys Workload Automation

Issue/Introduction

Autosys job (job name: test-job123) is schedule to run daily at 11:00pm. event_demon log had some error on 9/11/20, the job has "EVENT_HLDR_ERROR" error on 9/11/20 and since then test-job123 didn't run as schedule. We had to force start the job last night. 

[09/11/2020 22:22:00]     CAUAJM_E_18402 ORA-03135: connection lost contact

[09/11/2020 22:22:00]     Process ID: 23664

[09/11/2020 22:22:00]     Session ID: 1954 Serial number: 27745

[09/11/2020 22:22:00]     CAUAJM_E_18400 An error has occurred while interfacing with ORACLE.

[09/11/2020 22:22:00]     CAUAJM_W_10900 The database monitoring system has detected a potential problem with the database.

[09/11/2020 22:22:00]     CAUAJM_E_18401 Function <doExecute> invoked from <execute> failed <892>

[09/11/2020 22:22:00]     CAUAJM_I_10901 The database monitoring system is beginning validation of database connections.

[09/11/2020 22:22:00]     CAUAJM_I_10082 [test-machine123 connected for test-job234 1293.6543040.1]

[09/11/2020 22:22:00]     CAUAJM_I_10902 The database monitoring system has completed validation of database connections.

[09/11/2020 22:22:00]     CAUAJM_I_10903 The database monitoring system will resume monitoring for database problems.

[09/11/2020 22:22:01]     CAUAJM_I_10082 [test-machine123 connected for test-job345 1292.6543042.1]

[09/11/2020 22:22:01]     CAUAJM_I_10082 [test-machine34343 connected for test-job845 1294.6543041.1]

[09/11/2020 22:22:01]     CAUAJM_I_40245 EVENT: CHANGE_STATUS    STATUS: RUNNING         JOB: test-job345 MACHINE: test-machine123

[09/11/2020 22:22:01]     <Executing at AGENT>

[09/11/2020 22:22:03]     CAUAJM_E_18416 Event Server: <AUTOPRD>  Failed Query: <UPDATE ujo_job_status SET jc_pid='1' WHERE joid=1293 and job_ver=13>

[09/11/2020 22:22:03]     CAUAJM_E_18402 ORA-03135: connection lost contact

[09/11/2020 22:22:03]     Process ID: 23676

[09/11/2020 22:22:03]     Session ID: 668 Serial number: 12986

[09/11/2020 22:22:03]     CAUAJM_E_18400 An error has occurred while interfacing with ORACLE.

[09/11/2020 22:22:03]     CAUAJM_W_10900 The database monitoring system has detected a potential problem with the database.

[09/11/2020 22:22:03]     CAUAJM_E_18401 Function <doExecute> invoked from <execute> failed <908>

[09/11/2020 22:22:03]     CAUAJM_I_10901 The database monitoring system is beginning validation of database connections.

[09/11/2020 22:22:03]     CAUAJM_E_18416 Event Server: <AUTOPRD>  Failed Query: <BEGIN :RetVal := ujo_event_state (:I_eoid, :I_que_status, :I_evt_num); END; <<'ACE006cz3w00',2,22288393>>>

[09/11/2020 22:22:03]     CAUAJM_E_18402 ORA-03135: connection lost contact

[09/11/2020 22:22:03]     Process ID: 23687

[09/11/2020 22:22:03]     Session ID: 1633 Serial number: 59762

[09/11/2020 22:22:03]     CAUAJM_E_18400 An error has occurred while interfacing with ORACLE.

[09/11/2020 22:22:03]     CAUAJM_E_18401 Function <doExecute> invoked from <execute> failed <892>

[09/11/2020 22:22:03]     CAUAJM_I_10902 The database monitoring system has completed validation of database connections.

[09/11/2020 22:22:03]     CAUAJM_I_10903 The database monitoring system will resume monitoring for database problems.

[09/11/2020 22:22:03]     CAUAJM_W_10900 The database monitoring system has detected a potential problem with the database.

[09/11/2020 22:22:03]     CAUAJM_I_10901 The database monitoring system is beginning validation of database connections.

[09/11/2020 22:22:03]     CAUAJM_I_10902 The database monitoring system has completed validation of database connections.

[09/11/2020 22:22:03]     CAUAJM_I_10903 The database monitoring system will resume monitoring for database problems.

[09/11/2020 22:22:10]     CAUAJM_I_10165 The previously failed query below succeeded after 2 attempt(s):

[09/11/2020 22:22:10]     CAUAJM_I_18030 Event Server: <AUTOPRD>  Completed Stored Procedure: <BEGIN :RetVal := ujo_event_state (:I_eoid, :I_que_status, :I_evt_num); END; <<'ACE006cz3o00',2,22288393>>>

[09/11/2020 22:22:11]     CAUAJM_I_40245 EVENT: CHANGE_STATUS    STATUS: RUNNING         JOB: test-job845 MACHINE: test-machine34343

[09/11/2020 22:22:11]     <Executing at AGENT>

[09/11/2020 22:22:13]     CAUAJM_I_10165 The previously failed query below succeeded after 2 attempt(s):

[09/11/2020 22:22:13]     CAUAJM_I_18030 Event Server: <AUTOPRD>  Completed Stored Procedure: <BEGIN :RetVal := ujo_event_state (:I_eoid, :I_que_status, :I_evt_num); END; <<'ACE006cz3w00',2,22288393>>>

[09/11/2020 22:22:13]     CAUAJM_I_40245 EVENT: CHANGE_STATUS    STATUS: RUNNING         JOB: test-job234 MACHINE: test-machine123

[09/11/2020 22:22:13]     <Executing at AGENT>

[09/11/2020 22:22:13]     CAUAJM_I_10165 The previously failed query below succeeded after 2 attempt(s):

[09/11/2020 22:22:13]     CAUAJM_I_18031 Event Server: <AUTOPRD>  Completed Query: <UPDATE ujo_job_status SET jc_pid='1' WHERE joid=1293 and job_ver=13>

[09/11/2020 22:22:14]     CAUAJM_I_40245 EVENT: CHANGE_STATUS    STATUS: SUCCESS         JOB: test-job999  MACHINE: test-agent32344  EXITCODE:  0

 


[09/11/2020 22:30:00]     CAUAJM_E_18416 Event Server: <AUTOPRD>  Failed Query: <BEGIN :RetVal := ujo_event_state (:I_eoid, :I_que_status, :I_evt_num); END;>
[09/11/2020 22:30:00]     CAUAJM_E_18402 ORA-01403: no data found
[09/11/2020 22:30:00]     CAUAJM_E_18400 An error has occurred while interfacing with ORACLE.
[09/11/2020 22:30:00]     CAUAJM_E_18401 Function <doBind> invoked from <ibind> failed <818>
[09/11/2020 22:30:00]     CAUAJM_E_18400 An error has occurred while interfacing with ORACLE.
[09/11/2020 22:30:00]     CAUAJM_E_18401 Function <doParse> invoked from <parseInternal> failed <504>

[09/11/2020 23:00:01]     CAUAJM_I_40245 EVENT: ALARM            ALARM: EVENT_HDLR_ERROR JOB: test-job123
[09/11/2020 23:00:01]     <An error occurred while processing event <ACE006cpzx00> for job [test-job123 1057.6535465.0].>

Environment

Release : 11.3.6

Component : CA Workload Automation AE (AutoSys)

Resolution

Error in processing job event resulted in job was not scheduled for next run and had to be started manually.