Database Errors in event_demon log result in AutoSys shutdown

Products

Autosys Workload Automation

Issue/Introduction

Need help to understand an issue that resulted in AutoSys shutdown. Many DB related errors are noted in the event_demon log.

[10/01/2025 01:44:12] CAUAJM_E_18600 Database function call <SQLPrepare> failed in <parseInternal> with return code: -1[10/01/2025 01:44:03] CAUAJM_E_18802 Error from SQLExecute() Failed with SQL_ERROR.
[10/01/2025 01:44:03] CAUAJM_E_18601 SQLSTATE: 08S01, Native error: 10,060, Message: [Microsoft][ODBC Driver 18 for SQL Server]TCP Provider: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

[10/01/2025 02:51:18] CAUAJM_I_40315 Cross Platform Interface processing terminated.
[10/01/2025 02:51:18] CAUAJM_I_40247 AutoSys scheduler processing of events complete.
[10/01/2025 02:51:18] CAUAJM_I_40248 AutoSys scheduler shutdown complete. Exiting.

Cause

Noted, in event_demon logs, the initial error that led up to the above noted errors and of most significance is

[10/01/2025 01:44:01] CAUAJM_E_18601 SQLSTATE: 08S01, Native error: 10,060, Message: [Microsoft][ODBC Driver 18 for SQL Server]TCP Provider: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

This error indicates a network connectivity problem between an AutoSys component and the SQL Server database. This is the root of the problem. Following this error, is where the CAUAJM_E_18600, CAUAJM_E_18802 errors are noted.

Shifting focus to the as_server logs, around the same time, the following errors are thrown, which; again, are supportive of a communications related issue.

[10/01/2025 01:44:03] CAUAJM_E_18802 Error from SQLExecute() Failed with SQL_ERROR.
[10/01/2025 01:44:03] CAUAJM_E_18601 SQLSTATE: 08S01, Native error: 10,060, Message: [Microsoft][ODBC Driver 18 for SQL Server]TCP Provider: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

[10/01/2025 01:44:03] CAUAJM_E_18611 Event Server: <AUTOSYS-EVENT-SERVER,5150:AEDB> Failed Query: <SELECT int_val FROM ujo_alamode WHERE type='SEC'>
[10/01/2025 01:44:03] CAUAJM_W_10900 The database monitoring system has detected a potential problem with the database.
[10/01/2025 01:44:03] CAUAJM_I_10901 The database monitoring system is beginning validation of database connections.
[10/01/2025 01:44:03] CAUAJM_I_10902 The database monitoring system has completed validation of database connections.
[10/01/2025 01:44:03] CAUAJM_I_10903 The database monitoring system will resume monitoring for database problems.

Error Breakdown
This error can be broken down into the following components:

CAUAJME18601: This is an AutoSys error code that signifies a problem with a database operation.
SQLSTATE: 08S01: This is a standard ODBC error code indicating a communication link failure.
Native error: 10,060: This is a Microsoft SQL Server specific error code that points to a connection timeout.
Message: [Microsoft][ODBC Driver 18 for SQL Server]TCP Provider: A connection attempt failed…: This part of the message explicitly states that a connection could not be established over the network (TCP) because the database server did not respond in a timely manner.

Significance of the Error
This error signifies that the AutoSys application (such as the Application Server or Scheduler) is unable to communicate with its database. This is a critical error that can lead to various issues, including:

Jobs getting stuck in the "STARTING" position.
Slow response times and intermittent timeouts in the Web UI.
The shutdown of AutoSys services.

Common Causes
The primary cause of this error is network-related issues. These can include:

Network Latency: High latency between the AutoSys server and the database server, which can be common when moving from an on-premise environment to the cloud.
Network Packet Drop: Loss of data packets during transmission between the client and server.
Firewall or Port Issues: The correct port for the SQL Server instance may not be open or could be blocked on the network.
Incorrect Network Interface Card (NIC) Configuration: Issues with the network card drivers or settings.
DNS Resolution Problems: The hostname of the database server may not be resolvable from the AutoSys server.

Related Errors
You may also see other related error messages in the AutoSys logs that point towards a network or database connectivity problem:

CAUAJM_E_18600 Database function call <SQLDriverConnect> failed.
CAUAJM_E_10649 Server <hostname> was not available during connection operation.
[Microsoft][SQL Server Native Client 11.0]TCP Provider: An existing connection was forcibly closed by the remote host.
[Microsoft][SQL Server Native Client 11.0]TCP Provider: The semaphore timeout period has expired.
[Microsoft][SQL Server Native Client 11.0]Communication link failure.

Resolution

To resolve this issue, you should investigate the network connectivity between the AutoSys server and the database server. It is recommended to involve your network administration team.

Here are some steps to consider:

Verify Network Connectivity:
- Check if the database server's hostname is resolvable from the AutoSys server.
- Ensure that the correct port is being used and that it is not blocked by a firewall.
Investigate Network Performance:
- Work with your network team to check for any packet loss or high latency.
- Review the configuration of network adapters.
Tune AutoSys Connectivity:
- You may be able to improve the connectivity from the AutoSys manager to the database by increasing the DB_CONNECTIONS value in the configuration.
  - Modifying the DB_CONNECTIONS environment variable for Linux/UNIX occurs in the autosys $AUTOUSER/autosys.<shell>.<hostname> file in the format DB_CONNECTIONS=max_number.
    In conjunction with this setting, you'll also need to set SCHED_SCALE=limitExample:export DB_CONNECTIONS=45
    export SCHED_SCALE=10
  - On Windows, the SCHED_SCALE and DB_CONNECTIONS parameters are registry keys.