Autosys scheduler and application services stopped
Article ID: 248293
Updated On:
Products
CA Workload Automation AE
Issue/Introduction
Autosys services, scheduler (event_demon) and Application Server (as_server) were shut down in the operating system. The as_server log under $AUTOUSER/out directory has the following message captured.
[14/08/2022 03:40:13] CAUAJM_I_30031 Client [CA WAAE API JNI:2441][14][ip-10-01-01-01.schedule-server.internal:49451:10.01.01.01] [0x24151490][14/08/2022 03:40:13.5913][1:svc_autosys@APPS<@eemserver.schedule-server.internal> 1] API ID [299] execution started.
[14/08/2022 03:40:13] CAUAJM_I_30032 Client [CA WAAE API JNI:2441][14][ip-10-01-01-01.schedule-server.internal:49451:10.01.01.01] [0x24151490][14/08/2022 03:40:13.5913][1:svc_autosys@APPS<@eemserver.schedule-server.internal> 1] API ID [299] execution completed. Total time: 0.024 seconds.
[14/08/2022 03:40:15] CAUAJM_E_18416 Event Server: <autosysdbserver> Failed Query: <SELECT (CASE WHEN hostname='ip-10-01-01-01.schedule-server.internal' THEN 0 ELSE 1 END), hostname, mgr_alias, comm_alias, ma_status_id, time_stamp from ujo_ma_process WHERE (hostname='ip-10-01-01-01.schedule-server.internal') OR (hostname!='ip-10-01-01-01.schedule-server.internal' AND mgr_alias='APR_APP_autosys-app01.schedule-server.internal') OR (hostname!='ip-10-01-01-01.schedule-server.internal' AND comm_alias='APR_AUTOSYSAPP01') ORDER BY 1 ASC NULLS FIRST >
[14/08/2022 03:40:15] CAUAJM_E_18402 ORA-01089: immediate shutdown or close in progress - no operations are permitted
[14/08/2022 03:40:15] Process ID: 29899
[14/08/2022 03:40:15] Session ID: 2311 Serial number: 25447
[14/08/2022 03:40:15] CAUAJM_E_18400 An error has occurred while interfacing with ORACLE.
[14/08/2022 03:40:15] CAUAJM_E_18401 Function <doExecute> invoked from <execute> failed <908>
[14/08/2022 03:40:15] CAUAJM_W_10900 The database monitoring system has detected a potential problem with the database.
[14/08/2022 03:40:15] CAUAJM_I_10901 The database monitoring system is beginning validation of database connections.
[14/08/2022 03:40:15] CAUAJM_E_18416 Event Server: <autosysdbserver> Failed Query: <update ujo_alamode set str_val = (select str_val from ujo_alamode where type='AUTOSERV') where type = 'AUTOSERV'>
[14/08/2022 03:40:15] CAUAJM_E_18402 ORA-01089: immediate shutdown or close in progress - no operations are permitted
[14/08/2022 03:40:15] Process ID: 29878
[14/08/2022 03:40:15] Session ID: 1171 Serial number: 39304
[14/08/2022 03:40:15] CAUAJM_E_18400 An error has occurred while interfacing with ORACLE.
[14/08/2022 03:40:15] CAUAJM_E_18401 Function <doExecute> invoked from <execute> failed <908>
[14/08/2022 03:40:15] CAUAJM_W_10631 Error with database <autosysdbserver>. Checking connection.
[14/08/2022 03:40:15] CAUAJM_W_10632 Attempting to reconnect to database <autosysdbserver>. Attempt number [1].
[14/08/2022 03:40:15] CAUAJM_E_18407 Failed to connect to Oracle server: autosysdbserver
[14/08/2022 03:40:15] CAUAJM_E_18400 An error has occurred while interfacing with ORACLE.
[14/08/2022 03:40:15] CAUAJM_E_18401 Function <Olog> invoked from <openProc> failed <160>
[14/08/2022 03:40:15] CAUAJM_E_18402 ORA-12537: TNS:connection closed
[14/08/2022 03:41:15] CAUAJM_W_10632 Attempting to reconnect to database <autosysdbserver>. Attempt number [2].
[14/08/2022 03:41:16] CAUAJM_E_18407 Failed to connect to Oracle server: autosysdbserver
[14/08/2022 03:41:16] CAUAJM_E_18400 An error has occurred while interfacing with ORACLE.
[14/08/2022 03:41:16] CAUAJM_E_18401 Function <Olog> invoked from <openProc> failed <160>
[14/08/2022 03:41:16] CAUAJM_E_18402 ORA-12541: TNS:no listener
.....
...
[14/08/2022 04:29:35] CAUAJM_W_10632 Attempting to reconnect to database <autosysdbserver>. Attempt number [50].
[14/08/2022 04:29:36] CAUAJM_E_18407 Failed to connect to Oracle server: autosysdbserver
[14/08/2022 04:29:36] CAUAJM_E_18400 An error has occurred while interfacing with ORACLE.
[14/08/2022 04:29:36] CAUAJM_E_18401 Function <Olog> invoked from <openProc> failed <160>
[14/08/2022 04:29:36] CAUAJM_E_18402 ORA-12541: TNS:no listener
[14/08/2022 04:29:36] CAUAJM_W_10634 Exceeded maximum number of database reconnection attempts.
[14/08/2022 04:29:36] CAUAJM_E_40317 Unable to communicate with any database. Shutting down.
[14/08/2022 04:29:36] CAUAJM_I_10902 The database monitoring system has completed validation of database connections.
[14/08/2022 04:29:36] CAUAJM_I_10903 The database monitoring system will resume monitoring for database problems.
[14/08/2022 04:29:36] CAUAJM_W_10900 The database monitoring system has detected a potential problem with the database.
[14/08/2022 04:29:36] CAUAJM_I_10901 The database monitoring system is beginning validation of database connections.
[14/08/2022 04:29:36] CAUAJM_I_10902 The database monitoring system has completed validation of database connections.
[14/08/2022 04:29:36] CAUAJM_I_10903 The database monitoring system will resume monitoring for database problems.
[14/08/2022 04:29:37] CAUAJM_I_10493 CA EEM security session terminated.
[14/08/2022 04:29:37] CAUAJM_I_30004 CA WAAE Application Server shutdown complete.
[14/08/2022 04:29:37] CAUAJM_I_30000 CA WAAE Application Server exiting.
Environment
Release : 11.3.6 and later
Component : Autosys Scheduler and application server
Cause
The Autosys database (oracle) was issued 'shutdown immediate' as per the log snippet above.
Scheduler and application server could not reconnect to the database in 50 attempts per configuration.
Resolution
Restart the Autosys services (scheduler and application server) after the DB is back online.
Contact the database administrator to find out the reasons for the shutdown if it is unplanned.
The database shutdown should not be issued while the application services are running. If the db needs to be brought down for maintenance, stop all the autosys services (scheduler, application and webserver services along with WCC if running) before the DB shutdown.
Feedback
Yes
No
Powered by
