JOB stuck in starting position

Products

Autosys Workload Automation

Issue/Introduction

I have a job stuck in starting status, unable to KILLJOB or change status to anything, this job is part of a box set that has 31 jobs in it all depending on the previous job, thus it is holding up the whole process.

I am also experiencing slow responses from autosyslog -e, it can take up to 5min before displaying any processes.

WCC also has slow response and intermittent time-outs.

Please see attached documents for screenshots of the jobs status.

Since Moving Autosys on-premise environment to cloud, we are facing latencies over autosys job processing where we find slowness in Web UI and Autosys application server

Environment

Release : 12.0

Cause

CAUAJM_E_18601 SQLSTATE: IM008, Native error: 0, Message: [Microsoft][SQL Server Native Client 11.0]Dialog failed
CAUAJM_E_18601 SQLSTATE: 08S01, Native error: 10�054, Message: [Microsoft][SQL Server Native Client 11.0]TCP Provider: An existing connection was forcibly closed by the remote host.
CAUAJM_E_18601 SQLSTATE: 08S01, Native error: 121, Message: [Microsoft][SQL Server Native Client 11.0]TCP Provider: The semaphore timeout period has expired.
CAUAJM_E_18601 SQLSTATE: 08S01, Native error: 0, Message: [Microsoft][SQL Server Native Client 11.0]Communication link failure
CAUAJM_E_18601 SQLSTATE: IM008, Native error: 0, Message: [Microsoft][SQL Server Native Client 11.0]Dialog failed
CAUAJM_E_18601 SQLSTATE: 08S01, Native error: 0, Message: [Microsoft][SQL Server Native Client 11.0]Communication link failure
CAUAJM_E_18601 SQLSTATE: 08S01, Native error: 0, Message: [Microsoft][SQL Server Native Client 11.0]Communication link failure

CAUAJM_E_18601 SQLSTATE: 08S01, Native error: 121, Message: [Microsoft][SQL Server Native Client 11.0]TCP Provider: The semaphore timeout period has expired.

Causes Indicates:

There are various parameters that can affect network connectivity like network adaptors, packet drop, configured packet size etc. So i would suggest first you should connect to your network team and ask them to look into this issue. If network team has not found anything then you need to check application level network configurations at your level.

You should ask some questions to your network resource like whether any network packet drop being observed or not, or is there any NIC card issue etc. Let’s assume you have already reached out to network team and they haven’t found anything. I would suggest you to look into below options at your level to make sure everything is in place and not misconfigured.

Check TCP\IP settings
Ensure you have appropriate network packet size configured in SQL Server
Make sure you have properly configured TCP Chimney Offload
Validate you don’t have Network Interface Card (NIC) driver issue, you can get it checked with your network team

Resolution

Autosys Manager

Autosys Connectivity could be increased to improve connectivity from autosys manager to database.

AE Connectivity Tuning
Increase DB_CONNECTIONS value

Web UI

Deleted the WCC View that collects entire workload jobs from autosys manager which would impact on performance of WCC as it writes a huge query to database slows down collector processing

Web UI Best Practices

Based on Error Messages from Database, I suggest to consult DBAs and Network Admins to review the error messages and perform best practices to increase the connectivity

Below Questions Could be Inquired at Network Teams

There are various parameters that can affect network connectivity like network adaptors, packet drop, configured packet size etc. Should connect to respective network team and inquire them to look into this issue. If network team has not found anything then you need to check application level network configurations at your level.
You should ask some questions to your network resource like whether any network packet drop being observed or not, or is there any NIC card issue etc. Let’s assume you have already reached out to network team and they haven’t found anything. I would suggest you to look into below options at your level to make sure everything is in place and not misconfigured.

Check TCP\IP settings

Ensure you have appropriate network packet size configured in SQL Server
Check whether network team have properly configured TCP Chimney Offload
Validate not using Network Interface Card (NIC) driver, can get it checked with your network team