C_PERIOD fails if the last day of the month - can cause outage
search cancel

C_PERIOD fails if the last day of the month - can cause outage

book

Article ID: 210389

calendar_today

Updated On:

Products

CA Automic Workload Automation - Automation Engine

Issue/Introduction

The Jobs or Workflows running as recurring (C_PERIOD) can fail on last day of the month with FAULT_OTHER and stop executing.  This may also cause an outage on the last day of the month or first day of the month due to long running time critical calls and a system loop.

This happens when the following conditions are met:

  • The C_PERIOD is scheduled to skip certain days of the week (for instance Sundays or Saturdays and Sundays)
  • One of the skipped day is the last days of the month 

The issue is database independent and can cause different symptoms on different environments - for example it may be that job do not run during the time, it can be that sometimes there are no symptoms, or it can be that a loop occurs and there is a system outage.

Sometimes, the last Report will end on the before last day of the month with the following error: U01001200 Invalid input timestamp

It is possible variants of this scenario with from and to times going across midnight without any skipped days can also cause a system loop.  When this happens the following may be seen:

There will be very long running time critical database calls on statement:
select * from eh where eh_ah_idnr = ?

Sometimes this is accompanied by a message from Oracle:
U00003592 UCUDB - Status: '' Native error: '1555' Msg: 'ORA-01555: snapshot too old: rollback segment number 1 with name "snapshot_name" too small'
as well as locking on the db

 

The report log of the C_PERIOD might show also loop as shown below.

2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '00.00.0000 at '00:00' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:07' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:12' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:17' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:22' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:27' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:32' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:37' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:42' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:47' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:52' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:57' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:02' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:07' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:12' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:17' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:22' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:27' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:32' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:37' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:42' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '00.00.0000 at '00:00' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:07' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:12' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:17' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:22' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:27' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:32' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:37' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:42' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:47' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:52' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:57' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:02' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:07' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:12' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:17' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:22' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:27' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:32' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:37' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '01:42' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '00.00.0000 at '00:00' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:07' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:12' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:17' due to Server downtime.
2021-10-31 21:45:57 - U00011329 Task 'JOBP.BTA' did not start on '01.11.2021 at '00:22' due to Server downtime.

Environment

Release : 12.2 & 12.3

Component : AUTOMATION ENGINE

Cause

Improper calculation of date/time within the Automation Engine

Resolution

If during the time of the system looping (unable to login, no job running), a DBA is able to find a blocking session on the database, the blocking session can be killed and the system appears to recover at that point.

A COLD start sometimes helps with this situation, but understand that COLD starts can cause inconsistencies with running jobs and there may be additional recovery steps to get the jobs back to an optimal place.

The situation may resolve itself, but this could take anywhere from minutes to hours.

Solution:

Update to a fix version listed below or a newer version if available.


Fix version:

Component(s): Automation Engine

Automation.Engine 12.3.5 HF4 - available

Automation.Engine 12.3.6 - available

Automation.Engine 12.2.10+HF.2 - available

Powered by Wolken Software